Oct 30 05:30:12.359870 ip-10-0-130-194 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 05:30:12.828788 ip-10-0-130-194 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 05:30:12.828788 ip-10-0-130-194 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 05:30:12.828788 ip-10-0-130-194 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 05:30:12.828788 ip-10-0-130-194 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 05:30:12.837822 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837737    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:30:12.837822 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837818    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:30:12.837822 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837822    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:30:12.837822 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837825    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837829    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837831    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837834    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837837    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837840    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837844    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837848    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837852    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837855    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837861    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837864    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837866    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837870    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837872    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837876    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837878    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837881    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837883    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837886    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:30:12.837941 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837888    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837891    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837893    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837896    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837898    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837901    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837904    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837907    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837909    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837911    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837914    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837916    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837919    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837922    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837925    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837928    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837932    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837934    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837937    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837939    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:30:12.838403 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837942    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837945    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837949    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837953    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837956    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837959    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837961    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837964    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837966    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837969    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837971    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837973    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837976    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837978    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837981    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837983    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837985    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837988    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837992    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:30:12.838883 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837994    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837997    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.837999    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838001    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838004    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838006    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838009    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838012    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838015    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838018    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838021    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838024    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838026    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838029    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838031    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838034    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838036    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838038    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838041    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838043    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:30:12.839346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838046    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838048    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838051    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838053    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838055    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838058    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838060    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838063    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838065    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838068    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838072    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838074    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838077    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838079    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838082    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838084    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838087    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838091    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838093    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:30:12.840596 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838096    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838099    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838102    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838104    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838107    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838893    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838900    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838903    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838906    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838908    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838911    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838914    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838916    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838919    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838922    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838925    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838927    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838930    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838932    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838935    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:30:12.841050 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838937    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838940    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838942    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838945    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838948    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838950    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838953    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838955    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838958    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838960    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838963    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838965    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838968    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838971    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838973    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838976    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838979    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838981    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838985    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:30:12.841577 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838989    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838992    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838995    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.838997    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839000    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839002    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839005    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839007    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839010    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839012    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839014    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839017    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839019    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839022    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839025    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839028    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839030    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839033    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839035    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839038    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:30:12.842029 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839042    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839044    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839047    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839049    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839052    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839055    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839057    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839060    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839063    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839065    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839068    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839071    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839074    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839076    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839079    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839081    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839084    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839086    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839088    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:30:12.842559 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839091    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839093    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839096    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839098    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839100    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839103    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839105    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839108    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839110    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839113    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839116    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839118    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839120    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839123    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839127    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839130    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839133    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839136    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839139    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:30:12.843019 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839142    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839144    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839147    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839149    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839152    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839154    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839158    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839160    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839163    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839165    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839167    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839170    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839172    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839176    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839685    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839690    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839693    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839695    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839698    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:30:12.843474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839701    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839703    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839706    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839709    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839712    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839714    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839717    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839719    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839721    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839724    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839726    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839731    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839734    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839738    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839741    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839743    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839746    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839748    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839751    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839754    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:30:12.844038 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839756    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839759    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839762    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839764    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839766    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839769    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839772    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839774    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839778    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839781    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839784    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839786    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839789    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839791    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839794    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839796    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839799    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839801    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839806    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:30:12.844508 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839809    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839811    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839814    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839816    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839819    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839821    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839824    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839827    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839829    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839832    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839834    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839837    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839840    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839842    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839845    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839847    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839850    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839852    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839855    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839857    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:30:12.845002 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839860    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839862    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839864    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839867    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839869    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839872    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839875    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839877    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839880    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839882    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839885    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839890    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839893    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839895    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839898    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839900    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839903    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839905    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839908    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:30:12.845477 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839910    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839913    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839915    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839918    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839920    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839923    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839927    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839929    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839932    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839935    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839938    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839941    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839944    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839946    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839948    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839951    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839954    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839956    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839959    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:30:12.845935 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839961    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839963    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839966    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.839968    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840075    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840094    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840101    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840106    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840111    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840116    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840121    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840125    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840129    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840132    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840135    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840139    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840142    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840145    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840148    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840151    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840154    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840157    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840162    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 05:30:12.846379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840165    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840168    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840171    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840174    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840178    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840181    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840184    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840188    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840191    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840194    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840197    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840200    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840203    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840207    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840211    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840214    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840217    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840221    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840225    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840229    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840233    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840236    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840239    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840242    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 05:30:12.846956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840246    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840249    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840252    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840255    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840258    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840262    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840265    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840269    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840272    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840275    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840278    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840282    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840285    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840288    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840291    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840295    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840297    2579 flags.go:64] FLAG: --help="false"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840300    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-194.ec2.internal"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840303    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840306    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840310    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840313    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840317    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840320    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 05:30:12.847813 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840323    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840327    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840331    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840333    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840337    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840339    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840342    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840345    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840348    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840351    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840354    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840356    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840359    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840362    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840367    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840371    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840374    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840377    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840380    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840383    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840386    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840389    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840393    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840396    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840400    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 05:30:12.848682 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840403    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840407    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840410    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840413    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840416    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840421    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840424    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840431    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840434    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840438    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840441    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840444    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840447    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840450    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840453    2579 flags.go:64] FLAG: --port="10250"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840456    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840459    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08c24ae267f12a1dc"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840462    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840465    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840468    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840471    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840475    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840478    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840481    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840484    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840488    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 05:30:12.849277 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840491    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840494    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840497    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840499    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840502    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840505    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840508    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840511    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840514    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840517    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840520    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840537    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840541    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840544    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840547    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840551    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840554    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840557    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840560    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840565    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840568    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840571    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840575    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840578    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840581    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 05:30:12.849930 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840584    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840587    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840590    2579 flags.go:64] FLAG: --v="2"
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840594    2579 flags.go:64] FLAG: --version="false"
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840598    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840603    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.840607    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840746    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840751    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840754    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840757    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840759    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840762    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840772    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840774    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840777    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840780    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840782    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840786    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840789    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840791    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:30:12.850523 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840794    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840798    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840801    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840804    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840806    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840808    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840811    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840814    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840817    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840820    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840822    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840825    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840827    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840830    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840832    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840835    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840837    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840840    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840843    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:30:12.851046 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840846    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840848    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840851    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840853    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840856    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840858    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840861    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840863    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840866    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840868    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840872    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840874    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840877    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840879    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840883    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840886    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840888    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840890    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840893    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840895    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:30:12.851511 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840898    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840900    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840902    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840905    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840907    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840910    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840912    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840915    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840917    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840919    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840922    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840924    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840928    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840932    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840935    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840937    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840940    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840944    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840948    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:30:12.852036 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840951    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840954    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840957    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840960    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840963    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840965    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840968    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840971    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840974    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840976    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840979    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840982    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840984    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840987    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840989    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840992    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840994    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840996    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.840999    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841001    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:30:12.852482 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841004    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841006    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841009    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841011    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841014    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841017    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841020    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841022    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841024    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841027    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841029    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841032    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841034    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841037    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.841063    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 05:30:12.852983 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841106    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841111    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841114    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841117    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841120    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841123    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841125    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841128    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841131    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841133    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841136    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841138    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841141    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841143    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841145    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841148    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841150    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841153    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841155    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841158    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:30:12.853337 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841161    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841163    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841166    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841169    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841171    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841174    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841176    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841179    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841181    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841183    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841187    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841191    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841194    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841197    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841200    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841204    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841209    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841212    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841215    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:30:12.853817 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841218    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841221    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841223    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841225    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841228    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841230    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841233    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841235    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841237    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841240    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841243    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841245    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841248    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841250    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841253    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841255    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841257    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841261    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841263    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841266    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:30:12.854264 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841268    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841271    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841273    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841275    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841278    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841280    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841283    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841286    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841288    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841292    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841294    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841297    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841300    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841303    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841306    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841308    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841311    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841313    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841316    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:30:12.854754 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841318    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841320    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841323    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841325    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841327    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841330    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841332    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841335    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841337    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841340    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841342    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841345    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841348    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841350    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841353    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841355    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841358    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841360    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841363    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841366    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:30:12.855210 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841369    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841371    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841375    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841378    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841380    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841382    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841385    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.841389    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.849897    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.849914    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.850654    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.850661    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850723    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850728    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850731    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850734    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850736    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850739    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850743    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:30:12.856429 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850747    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850750    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850754    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850757    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850760    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850763    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850771    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850774    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850777    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850779    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850782    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850785    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850788    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850791    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850793    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850797    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850800    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850803    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850805    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:30:12.856907 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850808    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850811    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850818    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850821    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850824    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850827    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850829    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850833    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850837    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850840    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850843    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850845    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850848    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850850    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850853    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850855    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850858    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850860    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850863    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850865    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:30:12.858146 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850868    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850870    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850873    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850875    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850878    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850880    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850883    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850887    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850889    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850892    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850894    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850896    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850899    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850901    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850904    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850906    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850909    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850911    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850914    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850916    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:30:12.858658 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850919    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850921    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850923    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850926    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850928    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850931    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850933    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850936    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850939    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850941    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850944    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850946    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850949    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850952    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850954    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850957    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850959    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850962    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850965    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850967    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:30:12.859174 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850970    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850972    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850975    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850977    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850980    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850982    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850984    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850987    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850989    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850992    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850994    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850997    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.850999    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851002    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851004    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851007    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851009    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851011    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851014    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:30:12.859673 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851017    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.851045    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851188    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851193    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851196    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851199    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851201    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851205    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851207    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851210    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851212    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851215    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851219    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851224    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851227    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:30:12.860129 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851230    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851234    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851237    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851240    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851242    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851245    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851247    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851250    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851252    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851254    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851257    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851260    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851262    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851265    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851267    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851270    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851272    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851275    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851277    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851280    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:30:12.860474 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851283    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851285    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851288    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851290    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851293    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851296    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851299    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851301    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851304    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851307    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851310    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851312    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851315    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851317    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851320    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851322    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851325    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851327    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851331    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:30:12.860974 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851335    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851338    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851340    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851343    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851345    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851347    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851350    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851352    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851355    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851358    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851360    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851362    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851365    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851367    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851369    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851374    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851377    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851379    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851381    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:30:12.861444 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851385    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851387    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851390    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851392    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851395    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851397    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851400    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851402    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851405    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851408    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851410    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851413    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851415    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851418    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851420    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851423    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851425    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851427    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851430    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851433    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:30:12.862044 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851435    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851438    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851440    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851442    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851445    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851447    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851450    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851452    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851456    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851459    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851461    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851463    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851466    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851471    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851474    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.851498    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 05:30:12.862514 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851544    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851549    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851552    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851555    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851558    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851561    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851563    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851568    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851571    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851574    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851577    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851579    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851582    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851584    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851587    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851589    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851592    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851595    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851598    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:30:12.862910 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851600    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851602    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851605    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851608    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851610    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851613    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851616    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851618    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851621    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851624    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851626    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851630    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851633    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851635    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851638    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851640    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851642    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851645    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851647    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851650    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:30:12.863346 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851653    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851655    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851658    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851661    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851663    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851665    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851668    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851670    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851673    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851675    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851678    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851680    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851683    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851685    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851688    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851690    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851693    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851695    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851698    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851700    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:30:12.863833 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851702    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851705    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851708    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851710    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851713    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851717    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851721    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851724    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851727    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851730    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851732    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851735    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851739    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851741    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851744    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851746    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851748    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851751    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851754    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:30:12.864302 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851756    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851758    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851761    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851763    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851766    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851768    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851771    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851773    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851775    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851778    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851780    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851784    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851786    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851789    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851791    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851794    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851796    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851799    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851802    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851805    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:30:12.864763 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851807    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851809    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851812    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851814    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851817    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851820    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851823    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:12.851828    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.851880    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.853876    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.854883    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.854977    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.855022    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.857052    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 05:30:12.865226 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.858172    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 05:30:12.869939 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.869926    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 05:30:12.873837 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.873825    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 05:30:12.874722 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.874696    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 05:30:12.877659 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.877626    2579 fs.go:135] Filesystem UUIDs: map[139e0d9f-f0c0-41fd-9c44-a8fc2b553f61:/dev/nvme0n1p4 391513df-e5c0-4a32-b66e-64f1870c5773:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 05:30:12.877731 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.877656    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 05:30:12.885089 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.884960    2579 manager.go:217] Machine: {Timestamp:2026-10-30 05:30:12.88338005 +0000 UTC m=+0.389220732 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499996 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a2740e28994a7b33f2b2d05d33f3 SystemUUID:ec24a274-0e28-994a-7b33-f2b2d05d33f3 BootID:9f0c087d-d324-4152-9a0d-702ab9f066ec Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ef:af:29:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ef:af:29:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:db:7a:39:ad:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 05:30:12.885089 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.885082    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 05:30:12.885223 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.885165    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 05:30:12.888859 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.888832    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 05:30:12.889028 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.888860    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 05:30:12.889664 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.889652    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 05:30:12.889717 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.889669    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 05:30:12.889717 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.889712    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 05:30:12.889799 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.889733    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 05:30:12.891543 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.891515    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 05:30:12.895953 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.895939    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 05:30:12.896090 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.896079    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 05:30:12.897132 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.897119    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 05:30:12.897195 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.897138    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 05:30:12.897195 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.897153    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 05:30:12.897195 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.897165    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 05:30:12.897195 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.897178    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 05:30:12.897555 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.897541    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 05:30:12.900356 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.900340    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 05:30:12.906144 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.906131    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 05:30:12.907121 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.907108    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 05:30:12.907166 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.907139    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 05:30:12.907301 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.907290    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 05:30:12.907345 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.907312    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 05:30:12.907345 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.907322    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 05:30:12.907345 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.907333    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 05:30:12.907417 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.907345    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 05:30:12.907417 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.907359    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 05:30:12.908046 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.908035    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 05:30:12.908046 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.908046    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 05:30:12.908118 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.908055    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 05:30:12.908118 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.908062    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 05:30:12.908118 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.908076    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 05:30:12.908118 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.908084    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 05:30:12.908789 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.908779    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 05:30:12.908789 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.908789    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 05:30:12.912792 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.912779    2579 server.go:1263] "Started kubelet"
Oct 30 05:30:12.912890 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.912842    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 05:30:12.913250 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.912996    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 05:30:12.913376 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.913365    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 05:30:12.913974 ip-10-0-130-194 systemd[1]: Started Kubernetes Kubelet.
Oct 30 05:30:12.914344 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.914163    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 05:30:12.917711 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.917683    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 05:30:12.918968 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.918945    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 05:30:12.922229 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.922200    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 05:30:12.923406 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.923391    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 05:30:12.923517 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.923487    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found"
Oct 30 05:30:12.924015 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.923983    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 05:30:12.924015 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.923985    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 05:30:12.924126 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924036    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 05:30:12.924186 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924125    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 05:30:12.924186 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924134    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 05:30:12.924494 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924476    2579 factory.go:153] Registering CRI-O factory
Oct 30 05:30:12.924494 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924493    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 05:30:12.924638 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924572    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 05:30:12.924638 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924579    2579 factory.go:55] Registering systemd factory
Oct 30 05:30:12.924638 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924586    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 05:30:12.924638 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924602    2579 factory.go:103] Registering Raw factory
Oct 30 05:30:12.924638 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.924611    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 05:30:12.925509 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.925496    2579 manager.go:319] Starting recovery of all containers
Oct 30 05:30:12.936202 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.936188    2579 manager.go:324] Recovery completed
Oct 30 05:30:12.937943 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.937911    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 05:30:12.939605 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.939576    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 05:30:12.941007 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.940995    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 05:30:12.942698 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.942682    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 05:30:12.942742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.942710    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 05:30:12.942742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.942720    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID"
Oct 30 05:30:12.943288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.943272    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 05:30:12.943288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.943287    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 05:30:12.943412 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.943308    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 05:30:12.943412 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.943280    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 05:30:12.944048 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.944032    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 05:30:12.944104 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.944059    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 05:30:12.944104 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.944074    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID"
Oct 30 05:30:12.945508 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.945495    2579 policy_none.go:50] "Start"
Oct 30 05:30:12.945583 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.945511    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 05:30:12.945583 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.945521    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 05:30:12.947104 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.947087    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c7mpv"
Oct 30 05:30:12.947318 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.947302    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 05:30:12.947382 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.947311    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 05:30:12.947499 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.947488    2579 policy_none.go:44] "Start"
Oct 30 05:30:12.959024 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.957967    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-194.ec2.internal.18e3379a66cc65d7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-194.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-194.ec2.internal,},FirstTimestamp:2026-10-30 05:30:12.912752087 +0000 UTC m=+0.418592765,LastTimestamp:2026-10-30 05:30:12.912752087 +0000 UTC m=+0.418592765,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-194.ec2.internal,}"
Oct 30 05:30:12.959423 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.959409    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c7mpv"
Oct 30 05:30:12.971190 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.971170    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 05:30:12.971493 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.971476    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.993977    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.996603    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.997317    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.997342    2579 server.go:87] "Starting device plugin registration server"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.997613    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.997625    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.997737    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.997838    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:12.997847    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.998301    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:12.998346    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-194.ec2.internal\" not found"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.006161    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.006999    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.007566    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.007591    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.007614    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 05:30:13.013211 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:13.007652    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 05:30:13.023539 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.023510    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:30:13.098158 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.098138    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 05:30:13.098992 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.098975    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 05:30:13.099055 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.099005    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 05:30:13.099055 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.099017    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID"
Oct 30 05:30:13.099055 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.099040    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.107771 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.107744    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"]
Oct 30 05:30:13.107820 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.107803    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 05:30:13.108542 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.108514    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 05:30:13.108616 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.108562    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 05:30:13.108616 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.108573    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID"
Oct 30 05:30:13.110767 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.110756    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 05:30:13.110917 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.110895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.111012 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.110924    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 05:30:13.111460 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.111446    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 05:30:13.111562 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.111465    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 05:30:13.111562 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.111474    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID"
Oct 30 05:30:13.111562 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.111540    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 05:30:13.111719 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.111567    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 05:30:13.111719 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.111581    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID"
Oct 30 05:30:13.113685 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.113668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.113767 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.113698    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 05:30:13.114443 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.114425    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 05:30:13.114552 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.114452    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 05:30:13.114552 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.114461    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID"
Oct 30 05:30:13.120089 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.120076    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.120151 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:13.120095    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-194.ec2.internal\": node \"ip-10-0-130-194.ec2.internal\" not found"
Oct 30 05:30:13.124467 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.124453    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.125607 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.125589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.125689 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.125612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fd355e41b5bc4e012c326b7e91eb1cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"2fd355e41b5bc4e012c326b7e91eb1cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.125689 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.125629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.141144 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.141128    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.143800 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.143785    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.152131 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.152113    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 05:30:13.153304 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.153292    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.166489 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.166465    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 05:30:13.173173 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.173146    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 05:30:13.173267 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:13.173194    2579 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.190308 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.190294    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 05:30:13.190391 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:13.190323    2579 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.225933 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.225912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.225998 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.225941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.225998 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.225957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fd355e41b5bc4e012c326b7e91eb1cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"2fd355e41b5bc4e012c326b7e91eb1cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.225998 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.225990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fd355e41b5bc4e012c326b7e91eb1cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"2fd355e41b5bc4e012c326b7e91eb1cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.226122 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.225999    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.226122 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.226038    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.470364 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.470285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.477351 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.477336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:30:13.855110 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.855033    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 05:30:13.855558 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.855184    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 05:30:13.897886 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.897863    2579 apiserver.go:52] "Watching apiserver"
Oct 30 05:30:13.923282 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.923261    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 05:30:13.939707 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.939687    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 05:30:13.940021 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.940003    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-r6plj","kube-system/konnectivity-agent-6nsmc","openshift-cluster-node-tuning-operator/tuned-9k9f6","openshift-image-registry/node-ca-v25l9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal","openshift-multus/multus-rk729","openshift-multus/multus-additional-cni-plugins-sc5dg","openshift-multus/network-metrics-daemon-qtv7g","openshift-ovn-kubernetes/ovnkube-node-rm44x","kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal","openshift-dns/node-resolver-nlkm2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7","openshift-network-diagnostics/network-check-target-njsfm","openshift-network-operator/iptables-alerter-9wsmq"]
Oct 30 05:30:13.943189 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.943174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:13.945816 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.945797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:13.948458 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.948442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:13.951131 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.951113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:13.953769 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.953755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:13.956969 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.956954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:13.958018 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.958001    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 05:30:13.958293 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.958262    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 05:30:13.958375 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.958266    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 05:30:13.958375 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.958347    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wnglw\""
Oct 30 05:30:13.958375 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.958357    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 05:30:13.958488 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.958432    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 05:30:13.960096 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.960075    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 05:30:13.960096 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.960084    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 05:30:13.960238 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.960103    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-76x48\""
Oct 30 05:30:13.961597 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.961576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:13.961684 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:13.961656    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:13.962350 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.962332    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt4k5\""
Oct 30 05:30:13.962510 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.962480    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 05:30:13.962610 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.962509    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 05:30:13.964195 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.963982    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:25:14 +0000 UTC" deadline="2028-07-02 18:52:15.306381696 +0000 UTC"
Oct 30 05:30:13.964195 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.964027    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14677h22m1.342357985s"
Oct 30 05:30:13.964195 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.964175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:13.965917 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.965882    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 05:30:13.966426 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.965955    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 05:30:13.966517 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.966431    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w87jr\""
Oct 30 05:30:13.969192 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.965964    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 05:30:13.969339 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.969235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:13.969339 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:13.969308    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:13.969453 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.969426    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 05:30:13.969560 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.969501    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pts6t"
Oct 30 05:30:13.972290 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.972265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:13.974384 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.974357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:13.974481 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:13.974456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:13.976792 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.976773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rk729"
Oct 30 05:30:13.982353 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:13.982334    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pts6t"
Oct 30 05:30:14.025518 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.025498    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 05:30:14.030057 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81290ea5-5539-4ab9-9747-673e98c728d4-hosts-file\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.030160 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkskp\" (UniqueName: \"kubernetes.io/projected/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-kube-api-access-nkskp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.030160 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovn-node-metrics-cert\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.030241 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-binary-copy\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.030241 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:14.030311 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.030311 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.030382 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-dbus\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:14.030382 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmnv\" (UniqueName: \"kubernetes.io/projected/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-kube-api-access-2nmnv\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.030458 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdbbp\" (UniqueName: \"kubernetes.io/projected/3d421e45-b806-4fcb-b501-62bf9f4079c0-kube-api-access-cdbbp\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:14.030458 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-tmp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.030458 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-kubelet\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.030626 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-kubelet-config\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:14.030626 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:14.030626 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-system-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.030723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysconfig\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.030723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.030723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-host-slash\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.030827 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:14.030827 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-cnibin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.030827 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-etc-kubernetes\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.030827 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81290ea5-5539-4ab9-9747-673e98c728d4-tmp-dir\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.030827 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/087e82f9-024f-4bcd-97cf-8b7f928b719a-agent-certs\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:14.030999 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.030999 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-socket-dir-parent\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.030999 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-netns\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.030999 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-bin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.030999 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.030987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-etc-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031174 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.031174 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-iptables-alerter-script\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.031174 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-kubelet\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.031174 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-conf\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.031174 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-lib-modules\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.031378 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-systemd-units\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031378 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-env-overrides\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031378 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-hostroot\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.031378 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-daemon-config\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.031378 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-node-log\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031378 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.031378 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-config\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-cni-binary-copy\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-conf-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq48\" (UniqueName: \"kubernetes.io/projected/81290ea5-5539-4ab9-9747-673e98c728d4-kube-api-access-lwq48\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-slash\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-tuned\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-systemd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-bin\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8kh\" (UniqueName: \"kubernetes.io/projected/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kube-api-access-kf8kh\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.031743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/087e82f9-024f-4bcd-97cf-8b7f928b719a-konnectivity-ca\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-systemd\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-var-lib-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-ovn\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8chr\" (UniqueName: \"kubernetes.io/projected/51684f59-d0c8-4b0e-8f60-977be1864e8c-kube-api-access-f8chr\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-multus\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-kubernetes\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-host\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-log-socket\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.031993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-script-lib\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-multus-certs\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-modprobe-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/840883aa-27bd-49c8-a676-1d8336ec90c2-serviceca\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.032148 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-system-cni-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2c4\" (UniqueName: \"kubernetes.io/projected/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-kube-api-access-wx2c4\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-run\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-var-lib-kubelet\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840883aa-27bd-49c8-a676-1d8336ec90c2-host\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-netns\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-k8s-cni-cncf-io\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-sys\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6j95\" (UniqueName: \"kubernetes.io/projected/840883aa-27bd-49c8-a676-1d8336ec90c2-kube-api-access-c6j95\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-netd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-device-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cnibin\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.032742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.032630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7x2\" (UniqueName: \"kubernetes.io/projected/ab696c88-f337-4fd8-a9d7-be141c4c421c-kube-api-access-xw7x2\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.058714 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.058689    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 05:30:14.058799 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.058742    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 05:30:14.058973 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.058948    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 05:30:14.058973 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.058948    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 05:30:14.059297 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059279    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 05:30:14.059358 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059340    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-twkbr\""
Oct 30 05:30:14.059408 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059389    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 05:30:14.059408 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059400    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059404    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nnkr\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059400    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5cj6p\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059463    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059465    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059493    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059504    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059514    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-88sr6\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059560    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 05:30:14.059566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059568    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c6sjw\""
Oct 30 05:30:14.059869 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059589    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 05:30:14.059869 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059603    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 05:30:14.059869 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.059653    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 05:30:14.093876 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.093841    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd355e41b5bc4e012c326b7e91eb1cb.slice/crio-f1710ec37388ba46b5e004e7cea4f0b74eb72c88ae2b795deae975ac891d63a7 WatchSource:0}: Error finding container f1710ec37388ba46b5e004e7cea4f0b74eb72c88ae2b795deae975ac891d63a7: Status 404 returned error can't find the container with id f1710ec37388ba46b5e004e7cea4f0b74eb72c88ae2b795deae975ac891d63a7
Oct 30 05:30:14.094134 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.094109    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10023875eee95e78098f943c5150d3b0.slice/crio-0fae5a73bc0bec1c8fe0868046ec1d65af718f76d314d329413413eff7ff5d17 WatchSource:0}: Error finding container 0fae5a73bc0bec1c8fe0868046ec1d65af718f76d314d329413413eff7ff5d17: Status 404 returned error can't find the container with id 0fae5a73bc0bec1c8fe0868046ec1d65af718f76d314d329413413eff7ff5d17
Oct 30 05:30:14.095766 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.095647    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:30:14.097696 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.097683    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:30:14.133666 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-config\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.133666 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-cni-binary-copy\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.133666 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-conf-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.133819 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq48\" (UniqueName: \"kubernetes.io/projected/81290ea5-5539-4ab9-9747-673e98c728d4-kube-api-access-lwq48\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.133819 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-slash\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.133819 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133749    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-conf-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.133819 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133749    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-slash\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-tuned\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-systemd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-bin\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133957    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133980    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-bin\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.133980    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8kh\" (UniqueName: \"kubernetes.io/projected/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kube-api-access-kf8kh\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/087e82f9-024f-4bcd-97cf-8b7f928b719a-konnectivity-ca\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134022    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-systemd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-systemd\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-var-lib-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-ovn\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8chr\" (UniqueName: \"kubernetes.io/projected/51684f59-d0c8-4b0e-8f60-977be1864e8c-kube-api-access-f8chr\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.134501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134085    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-systemd\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-multus\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-var-lib-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-kubernetes\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-host\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-log-socket\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-script-lib\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134275    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-multus\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134319    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-ovn\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134340    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-config\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134363    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-log-socket\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-multus-certs\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134384    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134270    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-cni-binary-copy\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134405    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-kubernetes\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.135285 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134451    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-multus-certs\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134460    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-modprobe-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134509    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-host\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/840883aa-27bd-49c8-a676-1d8336ec90c2-serviceca\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134587    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-system-cni-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134622    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-modprobe-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134626    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-system-cni-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2c4\" (UniqueName: \"kubernetes.io/projected/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-kube-api-access-wx2c4\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-run\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-var-lib-kubelet\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840883aa-27bd-49c8-a676-1d8336ec90c2-host\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-netns\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-k8s-cni-cncf-io\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134808    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-var-lib-kubelet\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136139 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-sys\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134849    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-sys\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6j95\" (UniqueName: \"kubernetes.io/projected/840883aa-27bd-49c8-a676-1d8336ec90c2-kube-api-access-c6j95\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-run\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134912    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-netns\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-netd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134932    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-device-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840883aa-27bd-49c8-a676-1d8336ec90c2-host\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134976    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-netd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.134980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cnibin\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135001    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-device-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135007    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-k8s-cni-cncf-io\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7x2\" (UniqueName: \"kubernetes.io/projected/ab696c88-f337-4fd8-a9d7-be141c4c421c-kube-api-access-xw7x2\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135016    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cnibin\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81290ea5-5539-4ab9-9747-673e98c728d4-hosts-file\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135137    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81290ea5-5539-4ab9-9747-673e98c728d4-hosts-file\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.136956 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkskp\" (UniqueName: \"kubernetes.io/projected/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-kube-api-access-nkskp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovn-node-metrics-cert\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-binary-copy\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-dbus\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:14.135283    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmnv\" (UniqueName: \"kubernetes.io/projected/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-kube-api-access-2nmnv\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbbp\" (UniqueName: \"kubernetes.io/projected/3d421e45-b806-4fcb-b501-62bf9f4079c0-kube-api-access-cdbbp\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:14.135358    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret podName:a581d4f8-dc97-47d9-a409-bd391c36e67f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:14.635321031 +0000 UTC m=+2.141161730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret") pod "global-pull-secret-syncer-r6plj" (UID: "a581d4f8-dc97-47d9-a409-bd391c36e67f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-tmp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-kubelet\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-kubelet-config\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135549    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-dbus\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-system-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.137650 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysconfig\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135601    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-kubelet\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135412    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135647    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135673    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-kubelet-config\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-host-slash\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135727    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-binary-copy\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:14.135744    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135771    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-host-slash\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:14.135798    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs podName:3d421e45-b806-4fcb-b501-62bf9f4079c0 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:14.635772977 +0000 UTC m=+2.141613648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs") pod "network-metrics-daemon-qtv7g" (UID: "3d421e45-b806-4fcb-b501-62bf9f4079c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135822    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/840883aa-27bd-49c8-a676-1d8336ec90c2-serviceca\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135824    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-system-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135864    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysconfig\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-cnibin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-etc-kubernetes\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138205 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81290ea5-5539-4ab9-9747-673e98c728d4-tmp-dir\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/087e82f9-024f-4bcd-97cf-8b7f928b719a-agent-certs\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135964    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-cnibin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.135988    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-socket-dir-parent\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136049    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-netns\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136065    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/087e82f9-024f-4bcd-97cf-8b7f928b719a-konnectivity-ca\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136083    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-netns\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-bin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-etc-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136125    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-etc-kubernetes\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136175    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-socket-dir-parent\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136157    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136239    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81290ea5-5539-4ab9-9747-673e98c728d4-tmp-dir\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136240    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-etc-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.138762 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-iptables-alerter-script\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-kubelet\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136210    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-bin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-conf\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-lib-modules\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136379    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-kubelet\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-systemd-units\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-env-overrides\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-hostroot\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-daemon-config\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136506    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-conf\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-node-log\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136651    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-lib-modules\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136692    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136708    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-systemd-units\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136722    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-iptables-alerter-script\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136778    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-node-log\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.139230 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.136786    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-hostroot\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.139723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.137101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-env-overrides\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.139723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.137187    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-daemon-config\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.139723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.137279    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.139723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.137399    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-script-lib\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.139723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.138067    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-tuned\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.139723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.138099    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-tmp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.139723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.138316    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovn-node-metrics-cert\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.139723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.138602    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/087e82f9-024f-4bcd-97cf-8b7f928b719a-agent-certs\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:14.176255 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.176232    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7x2\" (UniqueName: \"kubernetes.io/projected/ab696c88-f337-4fd8-a9d7-be141c4c421c-kube-api-access-xw7x2\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.176502 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.176473    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbbp\" (UniqueName: \"kubernetes.io/projected/3d421e45-b806-4fcb-b501-62bf9f4079c0-kube-api-access-cdbbp\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:14.176602 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.176585    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmnv\" (UniqueName: \"kubernetes.io/projected/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-kube-api-access-2nmnv\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.176777 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.176760    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq48\" (UniqueName: \"kubernetes.io/projected/81290ea5-5539-4ab9-9747-673e98c728d4-kube-api-access-lwq48\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.177078 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.177060    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2c4\" (UniqueName: \"kubernetes.io/projected/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-kube-api-access-wx2c4\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.177737 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.177720    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6j95\" (UniqueName: \"kubernetes.io/projected/840883aa-27bd-49c8-a676-1d8336ec90c2-kube-api-access-c6j95\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.178205 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:14.178190    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 05:30:14.178252 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:14.178208    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 05:30:14.178252 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:14.178219    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4q7vq for pod openshift-network-diagnostics/network-check-target-njsfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:14.178334 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:14.178276    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq podName:b90d7ebe-df62-49e8-97e7-103ba14cd129 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:14.678263388 +0000 UTC m=+2.184104075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4q7vq" (UniqueName: "kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq") pod "network-check-target-njsfm" (UID: "b90d7ebe-df62-49e8-97e7-103ba14cd129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:14.179334 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.179317    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8kh\" (UniqueName: \"kubernetes.io/projected/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kube-api-access-kf8kh\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.182723 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.182705    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkskp\" (UniqueName: \"kubernetes.io/projected/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-kube-api-access-nkskp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.182954 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.182937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8chr\" (UniqueName: \"kubernetes.io/projected/51684f59-d0c8-4b0e-8f60-977be1864e8c-kube-api-access-f8chr\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.290523 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.290499    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:30:14.294290 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.294272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:30:14.301332 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.301313    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b35bb9_7f5b_4009_a6cf_15acaf83b02f.slice/crio-f0b6012dc0cee2e0173d1c27ac78d3970fb85701d1e9cf7074195dd710a26d9c WatchSource:0}: Error finding container f0b6012dc0cee2e0173d1c27ac78d3970fb85701d1e9cf7074195dd710a26d9c: Status 404 returned error can't find the container with id f0b6012dc0cee2e0173d1c27ac78d3970fb85701d1e9cf7074195dd710a26d9c
Oct 30 05:30:14.303324 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.303300    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81290ea5_5539_4ab9_9747_673e98c728d4.slice/crio-cf928504a563cea3e752b24a491b5a8ca88db0c13e126d1fe38998d0c6c1e6d5 WatchSource:0}: Error finding container cf928504a563cea3e752b24a491b5a8ca88db0c13e126d1fe38998d0c6c1e6d5: Status 404 returned error can't find the container with id cf928504a563cea3e752b24a491b5a8ca88db0c13e126d1fe38998d0c6c1e6d5
Oct 30 05:30:14.312773 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.312755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:30:14.316591 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.316419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:30:14.322789 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.322752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:30:14.323312 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.323291    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd29a4f1_41e1_4fb2_aef3_2b0abf8f8bae.slice/crio-aba7002911711a52bb10702fb399aa753ddf116adeb476b9b36f1adb7961a8fe WatchSource:0}: Error finding container aba7002911711a52bb10702fb399aa753ddf116adeb476b9b36f1adb7961a8fe: Status 404 returned error can't find the container with id aba7002911711a52bb10702fb399aa753ddf116adeb476b9b36f1adb7961a8fe
Oct 30 05:30:14.326960 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.326938    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840883aa_27bd_49c8_a676_1d8336ec90c2.slice/crio-6e6bf8f2095624f2453cd1741d8801b5838db0d6f7ae237041956acbb352ce0d WatchSource:0}: Error finding container 6e6bf8f2095624f2453cd1741d8801b5838db0d6f7ae237041956acbb352ce0d: Status 404 returned error can't find the container with id 6e6bf8f2095624f2453cd1741d8801b5838db0d6f7ae237041956acbb352ce0d
Oct 30 05:30:14.332247 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.332222    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdbc4e4_f579_4d7b_abc3_295b81e4f468.slice/crio-86e0d6284c01e3ba8246caab4a79288509c1e8f3e698088c2891223c178048ce WatchSource:0}: Error finding container 86e0d6284c01e3ba8246caab4a79288509c1e8f3e698088c2891223c178048ce: Status 404 returned error can't find the container with id 86e0d6284c01e3ba8246caab4a79288509c1e8f3e698088c2891223c178048ce
Oct 30 05:30:14.372655 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.372634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:30:14.375770 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.375753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:14.382450 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.382431    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb28f6d_9926_4b6a_b37a_149ae4785e1a.slice/crio-fe52fdd1efdf4085b572c56f940fd14235d8cb1be3a3e84459df6b6c164ad45f WatchSource:0}: Error finding container fe52fdd1efdf4085b572c56f940fd14235d8cb1be3a3e84459df6b6c164ad45f: Status 404 returned error can't find the container with id fe52fdd1efdf4085b572c56f940fd14235d8cb1be3a3e84459df6b6c164ad45f
Oct 30 05:30:14.384483 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.384462    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51684f59_d0c8_4b0e_8f60_977be1864e8c.slice/crio-62ed862a2a771f7c861fd0b52d87b5fcb9805e8259b4a1dd61b34fcd9d17255a WatchSource:0}: Error finding container 62ed862a2a771f7c861fd0b52d87b5fcb9805e8259b4a1dd61b34fcd9d17255a: Status 404 returned error can't find the container with id 62ed862a2a771f7c861fd0b52d87b5fcb9805e8259b4a1dd61b34fcd9d17255a
Oct 30 05:30:14.384569 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.384549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:14.391310 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:14.391297    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rk729"
Oct 30 05:30:14.393212 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.393194    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087e82f9_024f_4bcd_97cf_8b7f928b719a.slice/crio-bdd2baec5fd002bdb88c1a9db7ecd659bbd78a9a8accd057bd6eab1cd4203eb0 WatchSource:0}: Error finding container bdd2baec5fd002bdb88c1a9db7ecd659bbd78a9a8accd057bd6eab1cd4203eb0: Status 404 returned error can't find the container with id bdd2baec5fd002bdb88c1a9db7ecd659bbd78a9a8accd057bd6eab1cd4203eb0
Oct 30 05:30:14.400221 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:30:14.400196    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab696c88_f337_4fd8_a9d7_be141c4c421c.slice/crio-5b7c849b7b57f9a3ed10cfba9d508c1dd1982e5d378a7a467f057f695db60d52 WatchSource:0}: Error finding container 5b7c849b7b57f9a3ed10cfba9d508c1dd1982e5d378a7a467f057f695db60d52: Status 404 returned error can't find the container with id 5b7c849b7b57f9a3ed10cfba9d508c1dd1982e5d378a7a467f057f695db60d52
Oct 30 05:30:16.299992 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.299962    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:30:16.420566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.420491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:16.420566 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.420542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:16.420745 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:16.420722    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:16.420794 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:16.420784    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs podName:3d421e45-b806-4fcb-b501-62bf9f4079c0 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:17.420766203 +0000 UTC m=+3.146957507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs") pod "network-metrics-daemon-qtv7g" (UID: "3d421e45-b806-4fcb-b501-62bf9f4079c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:16.421191 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:16.421161    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:16.421268 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:16.421233    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret podName:a581d4f8-dc97-47d9-a409-bd391c36e67f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:17.421219025 +0000 UTC m=+3.147410331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret") pod "global-pull-secret-syncer-r6plj" (UID: "a581d4f8-dc97-47d9-a409-bd391c36e67f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:16.521396 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.521357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:16.521593 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:16.521569    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 05:30:16.521659 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:16.521598    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 05:30:16.521659 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:16.521612    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4q7vq for pod openshift-network-diagnostics/network-check-target-njsfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:16.521766 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:16.521670    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq podName:b90d7ebe-df62-49e8-97e7-103ba14cd129 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:17.521652561 +0000 UTC m=+3.247843852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4q7vq" (UniqueName: "kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq") pod "network-check-target-njsfm" (UID: "b90d7ebe-df62-49e8-97e7-103ba14cd129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:16.765663 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.765570    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:15 +0000 UTC" deadline="2028-06-23 16:19:37.508775707 +0000 UTC"
Oct 30 05:30:16.765663 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.765617    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14458h49m20.743160785s"
Oct 30 05:30:16.807005 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.806948    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6nsmc" event={"ID":"087e82f9-024f-4bcd-97cf-8b7f928b719a","Type":"ContainerStarted","Data":"bdd2baec5fd002bdb88c1a9db7ecd659bbd78a9a8accd057bd6eab1cd4203eb0"}
Oct 30 05:30:16.818170 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.818135    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"62ed862a2a771f7c861fd0b52d87b5fcb9805e8259b4a1dd61b34fcd9d17255a"}
Oct 30 05:30:16.824642 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.824570    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wsmq" event={"ID":"4bdbc4e4-f579-4d7b-abc3-295b81e4f468","Type":"ContainerStarted","Data":"86e0d6284c01e3ba8246caab4a79288509c1e8f3e698088c2891223c178048ce"}
Oct 30 05:30:16.836011 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.835922    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v25l9" event={"ID":"840883aa-27bd-49c8-a676-1d8336ec90c2","Type":"ContainerStarted","Data":"6e6bf8f2095624f2453cd1741d8801b5838db0d6f7ae237041956acbb352ce0d"}
Oct 30 05:30:16.844133 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.844104    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9k9f6" event={"ID":"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae","Type":"ContainerStarted","Data":"aba7002911711a52bb10702fb399aa753ddf116adeb476b9b36f1adb7961a8fe"}
Oct 30 05:30:16.864721 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.864685    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nlkm2" event={"ID":"81290ea5-5539-4ab9-9747-673e98c728d4","Type":"ContainerStarted","Data":"cf928504a563cea3e752b24a491b5a8ca88db0c13e126d1fe38998d0c6c1e6d5"}
Oct 30 05:30:16.883735 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.883421    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" event={"ID":"2fd355e41b5bc4e012c326b7e91eb1cb","Type":"ContainerStarted","Data":"f1710ec37388ba46b5e004e7cea4f0b74eb72c88ae2b795deae975ac891d63a7"}
Oct 30 05:30:16.890805 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.890748    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rk729" event={"ID":"ab696c88-f337-4fd8-a9d7-be141c4c421c","Type":"ContainerStarted","Data":"5b7c849b7b57f9a3ed10cfba9d508c1dd1982e5d378a7a467f057f695db60d52"}
Oct 30 05:30:16.896847 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.896794    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:30:16.911355 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.911290    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7" event={"ID":"adb28f6d-9926-4b6a-b37a-149ae4785e1a","Type":"ContainerStarted","Data":"fe52fdd1efdf4085b572c56f940fd14235d8cb1be3a3e84459df6b6c164ad45f"}
Oct 30 05:30:16.914569 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.914530    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerStarted","Data":"f0b6012dc0cee2e0173d1c27ac78d3970fb85701d1e9cf7074195dd710a26d9c"}
Oct 30 05:30:16.932851 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:16.932781    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"10023875eee95e78098f943c5150d3b0","Type":"ContainerStarted","Data":"0fae5a73bc0bec1c8fe0868046ec1d65af718f76d314d329413413eff7ff5d17"}
Oct 30 05:30:17.432114 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:17.432077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:17.432295 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:17.432189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:17.432347 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.432316    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:17.432391 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.432381    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret podName:a581d4f8-dc97-47d9-a409-bd391c36e67f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:19.432362787 +0000 UTC m=+5.158554079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret") pod "global-pull-secret-syncer-r6plj" (UID: "a581d4f8-dc97-47d9-a409-bd391c36e67f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:17.432788 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.432769    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:17.432863 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.432822    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs podName:3d421e45-b806-4fcb-b501-62bf9f4079c0 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:19.43280744 +0000 UTC m=+5.158998750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs") pod "network-metrics-daemon-qtv7g" (UID: "3d421e45-b806-4fcb-b501-62bf9f4079c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:17.533132 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:17.533095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:17.533328 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.533289    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 05:30:17.533328 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.533307    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 05:30:17.533328 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.533319    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4q7vq for pod openshift-network-diagnostics/network-check-target-njsfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:17.533489 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.533378    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq podName:b90d7ebe-df62-49e8-97e7-103ba14cd129 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:19.533359354 +0000 UTC m=+5.259550720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4q7vq" (UniqueName: "kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq") pod "network-check-target-njsfm" (UID: "b90d7ebe-df62-49e8-97e7-103ba14cd129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:17.766565 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:17.766430    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:15 +0000 UTC" deadline="2028-07-13 23:19:36.201713896 +0000 UTC"
Oct 30 05:30:17.766565 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:17.766466    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14945h49m18.435252663s"
Oct 30 05:30:17.789125 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:17.788759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:17.789125 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:17.788799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:17.789125 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.788883    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:17.789125 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.788946    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:17.789125 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:17.789006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:17.789125 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:17.789087    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:19.448160 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:19.448115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:19.448790 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:19.448247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:19.448790 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.448293    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:19.448790 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.448355    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:19.448790 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.448382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs podName:3d421e45-b806-4fcb-b501-62bf9f4079c0 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:23.448362121 +0000 UTC m=+9.174553424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs") pod "network-metrics-daemon-qtv7g" (UID: "3d421e45-b806-4fcb-b501-62bf9f4079c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:19.448790 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.448400    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret podName:a581d4f8-dc97-47d9-a409-bd391c36e67f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:23.448389447 +0000 UTC m=+9.174580744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret") pod "global-pull-secret-syncer-r6plj" (UID: "a581d4f8-dc97-47d9-a409-bd391c36e67f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:19.550214 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:19.549517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:19.550214 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.549756    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 05:30:19.550214 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.549778    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 05:30:19.550214 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.549791    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4q7vq for pod openshift-network-diagnostics/network-check-target-njsfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:19.550214 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.549846    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq podName:b90d7ebe-df62-49e8-97e7-103ba14cd129 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:23.549828827 +0000 UTC m=+9.276020131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4q7vq" (UniqueName: "kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq") pod "network-check-target-njsfm" (UID: "b90d7ebe-df62-49e8-97e7-103ba14cd129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:19.791420 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:19.791394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:19.791549 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.791526    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:19.791907 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:19.791888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:19.792032 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.792009    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:19.792103 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:19.792073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:19.792192 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:19.792142    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:19.971308 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:19.971068    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerStarted","Data":"a89038f6a50d996404d3fe2052092ad057143ee8e7e01a5560c7074bd7857cd3"}
Oct 30 05:30:19.974432 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:19.974000    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"10023875eee95e78098f943c5150d3b0","Type":"ContainerStarted","Data":"1e2c546e3032523d4fb6ef754b23fe43a37e3a8765dd126e556744eec339d917"}
Oct 30 05:30:20.978501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:20.978462    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" event={"ID":"2fd355e41b5bc4e012c326b7e91eb1cb","Type":"ContainerStarted","Data":"6a5023bfa0dd0b082a72d539ff0347852ae9e6ef3353a3f87ff64be82d501b0c"}
Oct 30 05:30:20.983160 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:20.983129    2579 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="a89038f6a50d996404d3fe2052092ad057143ee8e7e01a5560c7074bd7857cd3" exitCode=0
Oct 30 05:30:20.983333 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:20.983236    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerDied","Data":"a89038f6a50d996404d3fe2052092ad057143ee8e7e01a5560c7074bd7857cd3"}
Oct 30 05:30:20.988725 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:20.988699    2579 generic.go:356] "Generic (PLEG): container finished" podID="10023875eee95e78098f943c5150d3b0" containerID="1e2c546e3032523d4fb6ef754b23fe43a37e3a8765dd126e556744eec339d917" exitCode=0
Oct 30 05:30:20.988810 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:20.988790    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"10023875eee95e78098f943c5150d3b0","Type":"ContainerDied","Data":"1e2c546e3032523d4fb6ef754b23fe43a37e3a8765dd126e556744eec339d917"}
Oct 30 05:30:20.992975 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:20.992947    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6nsmc" event={"ID":"087e82f9-024f-4bcd-97cf-8b7f928b719a","Type":"ContainerStarted","Data":"0a4f44ac5102ecb46e26a5cdaa102f1edcc5356f611e6404cf9b6ae7c77bea97"}
Oct 30 05:30:21.105603 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:21.105542    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" podStartSLOduration=7.10552222 podStartE2EDuration="7.10552222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:21.035754782 +0000 UTC m=+6.761946089" watchObservedRunningTime="2026-10-30 05:30:21.10552222 +0000 UTC m=+6.831713531"
Oct 30 05:30:21.145849 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:21.145591    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6nsmc" podStartSLOduration=1.800256283 podStartE2EDuration="6.145574136s" totalImagesPullingTime="4.345317853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:21.14541457 +0000 UTC m=+6.871605880" watchObservedRunningTime="2026-10-30 05:30:21.145574136 +0000 UTC m=+6.871765447"
Oct 30 05:30:21.788444 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:21.788406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:21.788598 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:21.788537    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:21.788957 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:21.788936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:21.789079 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:21.789053    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:21.789151 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:21.789136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:21.789366 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:21.789347    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:21.998643 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:21.998604    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"10023875eee95e78098f943c5150d3b0","Type":"ContainerStarted","Data":"202599dc127fbe3808a3175498c48a312fa2f7d510fc14ebc32e1a22139dd484"}
Oct 30 05:30:22.002062 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:22.001648    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v25l9" event={"ID":"840883aa-27bd-49c8-a676-1d8336ec90c2","Type":"ContainerStarted","Data":"7dc2e31165a41b33c5c5bc1da38b980fd736d5f9ae88b132fcd48f09becf159e"}
Oct 30 05:30:22.079111 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:22.079054    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" podStartSLOduration=8.079035916 podStartE2EDuration="8.079035916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:22.038950714 +0000 UTC m=+7.765142026" watchObservedRunningTime="2026-10-30 05:30:22.079035916 +0000 UTC m=+7.805227227"
Oct 30 05:30:23.006683 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.006158    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7" event={"ID":"adb28f6d-9926-4b6a-b37a-149ae4785e1a","Type":"ContainerStarted","Data":"62b786c048677710946ad3512e1585b40911ff5fc5c0eba2ef57d336e20d6e6f"}
Oct 30 05:30:23.484819 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.484772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:23.484996 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.484826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:23.485053 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.485001    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:23.485107 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.485064    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs podName:3d421e45-b806-4fcb-b501-62bf9f4079c0 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:31.485044336 +0000 UTC m=+17.211235626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs") pod "network-metrics-daemon-qtv7g" (UID: "3d421e45-b806-4fcb-b501-62bf9f4079c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:23.485211 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.485138    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:23.485211 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.485172    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret podName:a581d4f8-dc97-47d9-a409-bd391c36e67f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:31.485161287 +0000 UTC m=+17.211352589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret") pod "global-pull-secret-syncer-r6plj" (UID: "a581d4f8-dc97-47d9-a409-bd391c36e67f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:23.585388 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.585343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:23.585510 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.585495    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 05:30:23.585559 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.585522    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 05:30:23.585559 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.585535    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4q7vq for pod openshift-network-diagnostics/network-check-target-njsfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:23.585644 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.585583    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq podName:b90d7ebe-df62-49e8-97e7-103ba14cd129 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:31.585566375 +0000 UTC m=+17.311757675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4q7vq" (UniqueName: "kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq") pod "network-check-target-njsfm" (UID: "b90d7ebe-df62-49e8-97e7-103ba14cd129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:23.739708 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.739643    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 05:30:23.782580 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.782470    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:30:23.739670841Z","UUID":"6e54d923-3928-4218-9b6e-720c8c23f1d1","Handler":null,"Name":"","Endpoint":""}
Oct 30 05:30:23.784089 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.784039    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 05:30:23.784089 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.784067    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 05:30:23.790392 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.790364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:23.790555 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.790531    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:23.790610 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.790572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:23.790684 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.790668    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:23.790749 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:23.790737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:23.790867 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:23.790850    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:24.010863 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:24.010785    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7" event={"ID":"adb28f6d-9926-4b6a-b37a-149ae4785e1a","Type":"ContainerStarted","Data":"0a0656ca1f235da9fbd6896f118de8cf9d2f688e72187d136a1a37d38bc7c4ac"}
Oct 30 05:30:24.466629 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:24.466592    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:24.468041 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:24.467816    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:24.510208 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:24.509885    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v25l9" podStartSLOduration=5.357055315 podStartE2EDuration="10.509863075s" totalImagesPullingTime="5.15280776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:22.079499446 +0000 UTC m=+7.805690762" watchObservedRunningTime="2026-10-30 05:30:24.509863075 +0000 UTC m=+10.236054386"
Oct 30 05:30:25.014298 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.014261    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nlkm2" event={"ID":"81290ea5-5539-4ab9-9747-673e98c728d4","Type":"ContainerStarted","Data":"e38da9281c61a4ad279e0a7bcacd0c3edb172a1595522737439d19e305188c92"}
Oct 30 05:30:25.015981 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.015950    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rk729" event={"ID":"ab696c88-f337-4fd8-a9d7-be141c4c421c","Type":"ContainerStarted","Data":"96de30d27dfc0c23fd43fa8066639f6b207a4f4f95d2b79a5307e41e0267f4e1"}
Oct 30 05:30:25.019396 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.019368    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7" event={"ID":"adb28f6d-9926-4b6a-b37a-149ae4785e1a","Type":"ContainerStarted","Data":"36b40846ea2d3e42d4ef280e12799e7a4554ffcf186023afebc4490208d5092d"}
Oct 30 05:30:25.019554 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.019428    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:25.019941 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.019921    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:30:25.097275 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.097217    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7" podStartSLOduration=2.813814973 podStartE2EDuration="11.097199609s" totalImagesPullingTime="8.283384636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:25.0971251 +0000 UTC m=+10.823316422" watchObservedRunningTime="2026-10-30 05:30:25.097199609 +0000 UTC m=+10.823390914"
Oct 30 05:30:25.098206 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.098147    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nlkm2" podStartSLOduration=2.783173467 podStartE2EDuration="11.098135597s" totalImagesPullingTime="8.31496213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:25.054304214 +0000 UTC m=+10.780495524" watchObservedRunningTime="2026-10-30 05:30:25.098135597 +0000 UTC m=+10.824326907"
Oct 30 05:30:25.140237 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.140152    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rk729" podStartSLOduration=2.045878654 podStartE2EDuration="10.140134384s" totalImagesPullingTime="8.09425573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:25.13967941 +0000 UTC m=+10.865870719" watchObservedRunningTime="2026-10-30 05:30:25.140134384 +0000 UTC m=+10.866325696"
Oct 30 05:30:25.788237 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.787549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:25.788237 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.787801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:25.788237 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:25.787760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:25.788237 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:25.787900    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:25.788237 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:25.787986    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:25.788237 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:25.788053    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:26.023691 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:26.023650    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wsmq" event={"ID":"4bdbc4e4-f579-4d7b-abc3-295b81e4f468","Type":"ContainerStarted","Data":"781950ebdc23e77b32ab779909c0af36918e47ba27d0277c568c9583eaf41293"}
Oct 30 05:30:26.026571 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:26.026547    2579 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="33e9fddc0f7dbd2cb056bd132f909f747fccf9baca9c047cd9bbef8dfa264516" exitCode=0
Oct 30 05:30:26.027658 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:26.027634    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerDied","Data":"33e9fddc0f7dbd2cb056bd132f909f747fccf9baca9c047cd9bbef8dfa264516"}
Oct 30 05:30:26.062998 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:26.062900    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9wsmq" podStartSLOduration=3.896779753 podStartE2EDuration="12.062879344s" totalImagesPullingTime="8.166099591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:26.062770548 +0000 UTC m=+11.788961857" watchObservedRunningTime="2026-10-30 05:30:26.062879344 +0000 UTC m=+11.789070653"
Oct 30 05:30:27.031673 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:27.031587    2579 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="b8dda2135827c434fb6cf79e3b51fe0e01b5076438b67f5f1888720fc0829799" exitCode=0
Oct 30 05:30:27.032190 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:27.031685    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerDied","Data":"b8dda2135827c434fb6cf79e3b51fe0e01b5076438b67f5f1888720fc0829799"}
Oct 30 05:30:27.033318 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:27.033292    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9k9f6" event={"ID":"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae","Type":"ContainerStarted","Data":"f6e9f5d4932f8424a2aa42e262dfd575e8ae44a0bb4afd499edd0f6c9d106c10"}
Oct 30 05:30:27.123026 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:27.122961    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9k9f6" podStartSLOduration=2.891066421 podStartE2EDuration="13.12294315s" totalImagesPullingTime="10.231876729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:27.122131387 +0000 UTC m=+12.848322697" watchObservedRunningTime="2026-10-30 05:30:27.12294315 +0000 UTC m=+12.849134461"
Oct 30 05:30:27.788210 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:27.788144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:27.788210 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:27.788170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:27.788429 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:27.788400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:27.788479 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:27.788454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:27.788579 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:27.788547    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:27.788707 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:27.788632    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:28.038045 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:28.038010    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerStarted","Data":"39b7e7c070db680b9794a5131bbb5865f7e65bf64e8a8af8abd846b395dec056"}
Oct 30 05:30:28.869016 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:28.868972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:29.043028 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:29.042912    2579 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="39b7e7c070db680b9794a5131bbb5865f7e65bf64e8a8af8abd846b395dec056" exitCode=0
Oct 30 05:30:29.043028 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:29.042976    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerDied","Data":"39b7e7c070db680b9794a5131bbb5865f7e65bf64e8a8af8abd846b395dec056"}
Oct 30 05:30:29.078730 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:29.078689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:29.787986 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:29.787954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:29.788164 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:29.788046    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:29.788412 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:29.788397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:29.788492 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:29.788473    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:29.788546 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:29.788523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:29.788602 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:29.788572    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:31.545347 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:31.544972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:31.545773 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.545117    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:31.545773 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:31.545369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:31.545773 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.545404    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs podName:3d421e45-b806-4fcb-b501-62bf9f4079c0 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:47.54539068 +0000 UTC m=+33.271581984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs") pod "network-metrics-daemon-qtv7g" (UID: "3d421e45-b806-4fcb-b501-62bf9f4079c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:31.545773 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.545483    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:31.545773 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.545545    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret podName:a581d4f8-dc97-47d9-a409-bd391c36e67f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:47.545530669 +0000 UTC m=+33.271721975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret") pod "global-pull-secret-syncer-r6plj" (UID: "a581d4f8-dc97-47d9-a409-bd391c36e67f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:31.640656 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:31.640623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:31.646623 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:31.646605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:31.646729 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.646716    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 05:30:31.646766 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.646734    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 05:30:31.646766 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.646743    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4q7vq for pod openshift-network-diagnostics/network-check-target-njsfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:31.646834 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.646783    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq podName:b90d7ebe-df62-49e8-97e7-103ba14cd129 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:47.646773025 +0000 UTC m=+33.372964312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4q7vq" (UniqueName: "kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq") pod "network-check-target-njsfm" (UID: "b90d7ebe-df62-49e8-97e7-103ba14cd129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:31.784661 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:31.784627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:31.787707 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:31.787694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:31.787776 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:31.787724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:31.787806 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:31.787699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:31.787806 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.787793    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:31.787877 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.787858    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:31.787960 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:31.787946    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:32.052508 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.052486    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:30:32.052832 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.052811    2579 generic.go:356] "Generic (PLEG): container finished" podID="51684f59-d0c8-4b0e-8f60-977be1864e8c" containerID="346fe73080056c3708496a92956c1fa391bb48b6b1dbb4dda2c62a77583f9844" exitCode=1
Oct 30 05:30:32.052940 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.052881    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"0bb880b1b0e5f4b718df5ef9738ec116e6aba2bb86a128e595a476055775e09e"}
Oct 30 05:30:32.052987 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.052938    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"1d1d2a0008896021d2b0bf129639139255ffa80ac61e9060647221fb25e80952"}
Oct 30 05:30:32.052987 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.052973    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"0aa739991bd97a26e670ac4616eac4e9f69f4dd2a26406383f4e1d78c2700d06"}
Oct 30 05:30:32.053050 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.053002    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"045e9c68b84c0b0f7a91e9abfe6542f74bce04b7c00d40ab90852fd4d43d625e"}
Oct 30 05:30:32.053050 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.053036    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerDied","Data":"346fe73080056c3708496a92956c1fa391bb48b6b1dbb4dda2c62a77583f9844"}
Oct 30 05:30:32.053111 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.053069    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"76c2001f1e8a067035378ea487b4a72343e299b4893af1d1d3b6a15b019d51cd"}
Oct 30 05:30:32.055603 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.055577    2579 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="e879ff480a7794b454c3c71fc6c6530310dd168b8eb5998c5ce49fe3a629c330" exitCode=0
Oct 30 05:30:32.055697 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:32.055616    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerDied","Data":"e879ff480a7794b454c3c71fc6c6530310dd168b8eb5998c5ce49fe3a629c330"}
Oct 30 05:30:33.060670 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:33.060634    2579 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="458f517e6142713aa4c2b17e27629a6e7b3a340306a310b63f3d818988fd7d30" exitCode=0
Oct 30 05:30:33.061018 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:33.060692    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerDied","Data":"458f517e6142713aa4c2b17e27629a6e7b3a340306a310b63f3d818988fd7d30"}
Oct 30 05:30:33.787638 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:33.787549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:33.787638 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:33.787589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:33.787638 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:33.787594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:33.787830 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:33.787668    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:33.787830 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:33.787780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:33.787891 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:33.787855    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:34.066153 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:34.066069    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" event={"ID":"64b35bb9-7f5b-4009-a6cf-15acaf83b02f","Type":"ContainerStarted","Data":"066865a834eca1c9fcdfbb0bf0a622a60fe27a4f7bd435f19b914b87a5456a08"}
Oct 30 05:30:34.068760 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:34.068741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:30:34.069137 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:34.069119    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"f55b5ad2e98cedffa8ea436641a2c7e7522d1cbe74264d573ea800b6062a6f8e"}
Oct 30 05:30:34.135305 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:34.135263    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sc5dg" podStartSLOduration=9.708391111 podStartE2EDuration="20.135252317s" totalImagesPullingTime="10.426861206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:34.134739284 +0000 UTC m=+19.860930594" watchObservedRunningTime="2026-10-30 05:30:34.135252317 +0000 UTC m=+19.861443626"
Oct 30 05:30:35.278233 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:35.278199    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:35.438507 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:35.438474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:35.585733 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:35.585710    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:35.763038 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:35.763009    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:35.788167 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:35.788102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:35.788167 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:35.788135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:35.788339 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:35.788265    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:35.788339 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:35.788310    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:35.788410 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:35.788355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:35.788449 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:35.788429    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:36.076317 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:36.076236    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:30:36.076576 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:36.076552    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"e1583d2258448a5fa9f640c2663fa7c642003ff3381a10635a237ef4bf21f674"}
Oct 30 05:30:36.076839 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:36.076811    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:36.076924 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:36.076909    2579 scope.go:122] "RemoveContainer" containerID="346fe73080056c3708496a92956c1fa391bb48b6b1dbb4dda2c62a77583f9844"
Oct 30 05:30:36.093469 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:36.093450    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:36.151623 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:36.151601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:36.385935 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:36.385759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:37.083091 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.083062    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:30:37.083458 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.083431    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" event={"ID":"51684f59-d0c8-4b0e-8f60-977be1864e8c","Type":"ContainerStarted","Data":"df160b87218434d716fc2fd1a1db8ba1769355d0afd79ab72f89e9ecebc2683c"}
Oct 30 05:30:37.083554 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.083542    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 05:30:37.083842 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.083825    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:37.099791 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.099771    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:37.165618 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.165587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:37.173885 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.173848    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" podStartSLOduration=8.357562421 podStartE2EDuration="23.173837204s" totalImagesPullingTime="14.816274783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:37.161373792 +0000 UTC m=+22.887565101" watchObservedRunningTime="2026-10-30 05:30:37.173837204 +0000 UTC m=+22.900028512"
Oct 30 05:30:37.315406 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.315363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:37.641469 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.641435    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njsfm"]
Oct 30 05:30:37.641927 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.641578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:37.641927 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:37.641688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:37.644516 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.644493    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r6plj"]
Oct 30 05:30:37.644636 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.644597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:37.644711 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:37.644691    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:37.676461 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.676438    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qtv7g"]
Oct 30 05:30:37.676565 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.676552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:37.676697 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:37.676670    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:37.853668 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:37.853639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:38.025760 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:38.025667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:38.085456 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:38.085429    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 05:30:38.206980 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:38.206950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:38.787727 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:38.787693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:38.788115 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:38.787818    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:38.788115 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:38.787860    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:38.788115 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:38.787945    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:39.087863 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:39.087792    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 05:30:39.417620 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:39.417589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:39.622630 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:39.622604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:39.784626 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:39.784599    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:39.788060 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:39.788042    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:39.788387 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:39.788141    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:39.985578 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:39.985492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:40.155812 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.155774    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:30:40.155994 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.155980    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 05:30:40.167163 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.167118    2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" podUID="51684f59-d0c8-4b0e-8f60-977be1864e8c" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 05:30:40.176146 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.176117    2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x" podUID="51684f59-d0c8-4b0e-8f60-977be1864e8c" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 05:30:40.179514 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.179488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:40.490475 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.490444    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:40.656323 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.656281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:40.788022 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.787954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:40.788022 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.787988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:40.788490 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:40.788069    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:40.788490 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:40.788221    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:40.818254 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:40.818234    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nlkm2_81290ea5-5539-4ab9-9747-673e98c728d4/dns-node-resolver/0.log"
Oct 30 05:30:41.392926 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:41.392895    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:41.592330 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:41.592286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:41.782826 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:41.782734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:41.787853 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:41.787828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:41.787957 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:41.787941    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:41.817613 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:41.817590    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v25l9_840883aa-27bd-49c8-a676-1d8336ec90c2/node-ca/0.log"
Oct 30 05:30:41.986653 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:41.986625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:42.185691 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:42.185663    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:42.507123 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:42.507038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:42.658947 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:42.658919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:42.788198 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:42.788102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:42.788343 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:42.788207    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:42.788343 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:42.788236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:42.788412 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:42.788357    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:42.806241 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:42.806216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:42.941927 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:42.941863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:43.346169 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:43.346136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:43.482698 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:43.482657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:43.623542 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:43.623515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:43.774661 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:43.774628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:43.787457 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:43.787431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:43.787580 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:43.787535    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:43.983634 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:43.983602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:44.200133 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:44.200102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:44.740306 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:44.740269    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:44.788466 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:44.788438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:44.788637 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:44.788603    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:44.788637 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:44.788608    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:44.788756 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:44.788688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:44.874754 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:44.874724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:45.115318 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:45.115238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:45.385041 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:45.385013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:45.577710 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:45.577681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:45.781449 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:45.781382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:45.788379 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:45.788357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:45.788472 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:45.788456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:45.990497 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:45.990462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:46.173648 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:46.173621    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:46.788257 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:46.788222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:46.788658 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:46.788335    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:46.788658 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:46.788396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:46.788658 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:46.788508    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:47.035571 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.035532    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:47.383849 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.383816    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:47.563038 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.563006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:47.563038 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.563050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:47.563316 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.563150    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:47.563316 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.563212    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:47.563316 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.563245    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret podName:a581d4f8-dc97-47d9-a409-bd391c36e67f nodeName:}" failed. No retries permitted until 2026-10-30 05:31:19.563224327 +0000 UTC m=+65.289415616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret") pod "global-pull-secret-syncer-r6plj" (UID: "a581d4f8-dc97-47d9-a409-bd391c36e67f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 05:30:47.563316 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.563269    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs podName:3d421e45-b806-4fcb-b501-62bf9f4079c0 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:19.56325275 +0000 UTC m=+65.289444044 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs") pod "network-metrics-daemon-qtv7g" (UID: "3d421e45-b806-4fcb-b501-62bf9f4079c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 05:30:47.584092 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.584066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:47.664261 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.664231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:47.664393 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.664371    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 05:30:47.664393 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.664388    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 05:30:47.664633 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.664400    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4q7vq for pod openshift-network-diagnostics/network-check-target-njsfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:47.664633 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.664455    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq podName:b90d7ebe-df62-49e8-97e7-103ba14cd129 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:19.664440075 +0000 UTC m=+65.390631381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4q7vq" (UniqueName: "kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq") pod "network-check-target-njsfm" (UID: "b90d7ebe-df62-49e8-97e7-103ba14cd129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 05:30:47.781138 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.781106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:47.788187 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.788161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:47.788290 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:47.788273    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:47.979488 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:47.979448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:48.185793 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:48.185761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:48.338906 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:48.338841    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:48.777322 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:48.777284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:48.788266 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:48.788238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:48.788266 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:48.788268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:48.788447 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:48.788351    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:48.788509 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:48.788464    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:48.986792 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:48.986758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:49.787971 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:49.787939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:49.788470 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:49.788063    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:50.793044 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:50.793015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:50.793520 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:50.793132    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:50.793520 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:50.793169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:50.793520 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:50.793276    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:51.788023 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:51.787990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:51.788204 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:51.788095    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:52.163165 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:52.163132    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:52.303587 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:52.303555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:52.788342 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:52.788312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:52.788556 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:52.788320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:52.788556 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:52.788421    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:52.788556 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:52.788486    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:53.788168 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:53.788139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:53.788589 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:53.788258    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:54.617594 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:54.617560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:54.762383 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:54.762351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:54.790542 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:54.790516    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:54.790542 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:54.790517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:54.790976 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:54.790628    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:54.790976 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:54.790709    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:55.788387 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:55.788360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:55.788528 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:55.788457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:56.788458 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:56.788424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:56.788919 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:56.788584    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:56.788919 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:56.788624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:56.788919 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:56.788688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:57.787814 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:57.787786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:57.787974 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:57.787883    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:30:58.013156 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:58.013116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:58.763872 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:58.763840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:58.787755 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:58.787729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:30:58.787879 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:58.787856    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsfm" podUID="b90d7ebe-df62-49e8-97e7-103ba14cd129"
Oct 30 05:30:58.787948 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:58.787930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:30:58.788055 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:58.788028    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r6plj" podUID="a581d4f8-dc97-47d9-a409-bd391c36e67f"
Oct 30 05:30:58.945914 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:58.945878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:59.082902 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:59.082818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:59.787427 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:30:59.787396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:30:59.787572 ip-10-0-130-194 kubenswrapper[2579]: E1030 05:30:59.787534    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qtv7g" podUID="3d421e45-b806-4fcb-b501-62bf9f4079c0"
Oct 30 05:31:00.015063 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.015001    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeReady"
Oct 30 05:31:00.015209 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.015102    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 05:31:00.120759 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.120732    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gszgm"]
Oct 30 05:31:00.123804 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.123788    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9665q"]
Oct 30 05:31:00.123919 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.123904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.126757 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.126738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:00.144856 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.144835    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9665q"]
Oct 30 05:31:00.147129 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.147107    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 05:31:00.147286 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.147132    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 05:31:00.147286 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.147160    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 05:31:00.147286 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.147218    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 05:31:00.147458 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.147312    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 05:31:00.147458 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.147390    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 05:31:00.147878 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.147858    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gszgm"]
Oct 30 05:31:00.155705 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.155688    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnf9b\""
Oct 30 05:31:00.158539 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.158520    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 05:31:00.158622 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.158539    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9q7xl\""
Oct 30 05:31:00.176087 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.176067    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rvnfl"]
Oct 30 05:31:00.178972 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.178958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.198382 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.198362    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rvnfl"]
Oct 30 05:31:00.247859 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.247799    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 05:31:00.248102 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.248085    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wzzmx\""
Oct 30 05:31:00.248204 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.248112    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 05:31:00.265500 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.265445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-data-volume\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.265500 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.265483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-crio-socket\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.265613 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.265513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlh8n\" (UniqueName: \"kubernetes.io/projected/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-api-access-vlh8n\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.265647 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.265617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.265678 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.265648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.265713 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.265681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d0ef442-4160-4ab1-80f1-03b37851a061-cert\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:00.265713 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.265703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2r6d\" (UniqueName: \"kubernetes.io/projected/3d0ef442-4160-4ab1-80f1-03b37851a061-kube-api-access-b2r6d\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:00.366803 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.366775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7bb4\" (UniqueName: \"kubernetes.io/projected/1e3b6130-4545-4c00-ab54-0c40c7a906bb-kube-api-access-b7bb4\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.366803 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.366808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlh8n\" (UniqueName: \"kubernetes.io/projected/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-api-access-vlh8n\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.366970 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.366826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e3b6130-4545-4c00-ab54-0c40c7a906bb-metrics-tls\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.366970 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.366847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.366970 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.366904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.366970 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.366935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d0ef442-4160-4ab1-80f1-03b37851a061-cert\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:00.366970 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.366953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2r6d\" (UniqueName: \"kubernetes.io/projected/3d0ef442-4160-4ab1-80f1-03b37851a061-kube-api-access-b2r6d\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:00.367234 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.367086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b6130-4545-4c00-ab54-0c40c7a906bb-config-volume\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.367234 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.367215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-data-volume\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.367336 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.367250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-crio-socket\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.367336 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.367272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b6130-4545-4c00-ab54-0c40c7a906bb-tmp-dir\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.367488 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.367468    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-crio-socket\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.367564 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.367550    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-data-volume\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.367627 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.367607    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.370556 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.370535    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.370652 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.370607    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d0ef442-4160-4ab1-80f1-03b37851a061-cert\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:00.389894 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.389863    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2r6d\" (UniqueName: \"kubernetes.io/projected/3d0ef442-4160-4ab1-80f1-03b37851a061-kube-api-access-b2r6d\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:00.392212 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.392174    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlh8n\" (UniqueName: \"kubernetes.io/projected/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-api-access-vlh8n\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.440501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.440470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:00.442526 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.442510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:00.467817 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.467791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b6130-4545-4c00-ab54-0c40c7a906bb-tmp-dir\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.467952 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.467830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7bb4\" (UniqueName: \"kubernetes.io/projected/1e3b6130-4545-4c00-ab54-0c40c7a906bb-kube-api-access-b7bb4\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.467952 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.467859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e3b6130-4545-4c00-ab54-0c40c7a906bb-metrics-tls\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.468142 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.468101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b6130-4545-4c00-ab54-0c40c7a906bb-config-volume\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.468488 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.468463    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b6130-4545-4c00-ab54-0c40c7a906bb-tmp-dir\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.468830 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.468810    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b6130-4545-4c00-ab54-0c40c7a906bb-config-volume\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.470217 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.470200    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e3b6130-4545-4c00-ab54-0c40c7a906bb-metrics-tls\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.491793 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.491764    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7bb4\" (UniqueName: \"kubernetes.io/projected/1e3b6130-4545-4c00-ab54-0c40c7a906bb-kube-api-access-b7bb4\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.766766 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.766713    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9665q"]
Oct 30 05:31:00.772188 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:00.772151    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0ef442_4160_4ab1_80f1_03b37851a061.slice/crio-4c44b9e34b90be5c951572b3193574cf979d3b7c87cf6b201c13305f8372fc09 WatchSource:0}: Error finding container 4c44b9e34b90be5c951572b3193574cf979d3b7c87cf6b201c13305f8372fc09: Status 404 returned error can't find the container with id 4c44b9e34b90be5c951572b3193574cf979d3b7c87cf6b201c13305f8372fc09
Oct 30 05:31:00.779291 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.779260    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gszgm"]
Oct 30 05:31:00.783871 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:00.783847    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180d488f_b7df_4f3b_a6fb_b3c69f264d5c.slice/crio-0ff467ebb2cfce1d0fa7a2aa50f84e935dd4f08a8f877b7dbb746050c6e1382a WatchSource:0}: Error finding container 0ff467ebb2cfce1d0fa7a2aa50f84e935dd4f08a8f877b7dbb746050c6e1382a: Status 404 returned error can't find the container with id 0ff467ebb2cfce1d0fa7a2aa50f84e935dd4f08a8f877b7dbb746050c6e1382a
Oct 30 05:31:00.791253 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.787938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:31:00.791253 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.787982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:00.791253 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.791020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:00.803677 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.803657    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c4j6h\""
Oct 30 05:31:00.806797 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.806773    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 05:31:00.806908 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.806780    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w4h88\""
Oct 30 05:31:00.806908 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.806849    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 05:31:00.806908 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:00.806863    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 05:31:01.087207 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.087120    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rvnfl"]
Oct 30 05:31:01.092211 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:01.092171    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3b6130_4545_4c00_ab54_0c40c7a906bb.slice/crio-1e9dbf9926bdb9e9741a66e8cbe7e0e2fb0438702fc2a99ec9db630a08183943 WatchSource:0}: Error finding container 1e9dbf9926bdb9e9741a66e8cbe7e0e2fb0438702fc2a99ec9db630a08183943: Status 404 returned error can't find the container with id 1e9dbf9926bdb9e9741a66e8cbe7e0e2fb0438702fc2a99ec9db630a08183943
Oct 30 05:31:01.136250 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.136221    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvnfl" event={"ID":"1e3b6130-4545-4c00-ab54-0c40c7a906bb","Type":"ContainerStarted","Data":"1e9dbf9926bdb9e9741a66e8cbe7e0e2fb0438702fc2a99ec9db630a08183943"}
Oct 30 05:31:01.137354 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.137328    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9665q" event={"ID":"3d0ef442-4160-4ab1-80f1-03b37851a061","Type":"ContainerStarted","Data":"4c44b9e34b90be5c951572b3193574cf979d3b7c87cf6b201c13305f8372fc09"}
Oct 30 05:31:01.138709 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.138690    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gszgm" event={"ID":"180d488f-b7df-4f3b-a6fb-b3c69f264d5c","Type":"ContainerStarted","Data":"c5066faf8085223844bad942c3274b6aecc4c522a7fbbe379bd7ca14127d5acf"}
Oct 30 05:31:01.138788 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.138729    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gszgm" event={"ID":"180d488f-b7df-4f3b-a6fb-b3c69f264d5c","Type":"ContainerStarted","Data":"0ff467ebb2cfce1d0fa7a2aa50f84e935dd4f08a8f877b7dbb746050c6e1382a"}
Oct 30 05:31:01.220913 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.220858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:01.290343 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.290306    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"]
Oct 30 05:31:01.293512 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.293465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.314569 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.314508    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 05:31:01.314569 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.314508    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fl95n\""
Oct 30 05:31:01.314768 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.314581    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 05:31:01.314911 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.314883    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 05:31:01.315029 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.314926    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 05:31:01.315101 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.315034    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 05:31:01.315742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.315718    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"]
Oct 30 05:31:01.373017 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.372928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:01.475383 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.475353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7c8g\" (UniqueName: \"kubernetes.io/projected/883d9833-be84-411b-998e-cea7379b6263-kube-api-access-b7c8g\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.475499 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.475467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.475499 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.475491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/883d9833-be84-411b-998e-cea7379b6263-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.475606 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.475518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.508168 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.508128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:01.576905 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.576876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7c8g\" (UniqueName: \"kubernetes.io/projected/883d9833-be84-411b-998e-cea7379b6263-kube-api-access-b7c8g\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.577044 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.576946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.577044 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.576987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/883d9833-be84-411b-998e-cea7379b6263-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.577044 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.577029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.578135 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.578106    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/883d9833-be84-411b-998e-cea7379b6263-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.580156 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.580108    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.582631 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.582604    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.600632 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.600607    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7c8g\" (UniqueName: \"kubernetes.io/projected/883d9833-be84-411b-998e-cea7379b6263-kube-api-access-b7c8g\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.608252 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.608233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:01.788402 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.787811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:31:01.802714 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.802654    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-897vl\""
Oct 30 05:31:01.802837 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.802694    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 05:31:01.806629 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.806607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:01.906997 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.906965    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"]
Oct 30 05:31:01.913053 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:01.913022    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883d9833_be84_411b_998e_cea7379b6263.slice/crio-55ddd6fa2f4f279b34fc1061da2f63bf24e4530c320964a81c7a91d18636d1ce WatchSource:0}: Error finding container 55ddd6fa2f4f279b34fc1061da2f63bf24e4530c320964a81c7a91d18636d1ce: Status 404 returned error can't find the container with id 55ddd6fa2f4f279b34fc1061da2f63bf24e4530c320964a81c7a91d18636d1ce
Oct 30 05:31:01.954028 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:01.953978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:02.144467 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:02.144429    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gszgm" event={"ID":"180d488f-b7df-4f3b-a6fb-b3c69f264d5c","Type":"ContainerStarted","Data":"2a42391975d44c984f5ceee9caa297b985c1e9783a8ddda4ff15059adbec0ee6"}
Oct 30 05:31:02.145947 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:02.145917    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf" event={"ID":"883d9833-be84-411b-998e-cea7379b6263","Type":"ContainerStarted","Data":"55ddd6fa2f4f279b34fc1061da2f63bf24e4530c320964a81c7a91d18636d1ce"}
Oct 30 05:31:02.898335 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:02.898286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:03.040533 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:03.040506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:03.151941 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:03.151909    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gszgm" event={"ID":"180d488f-b7df-4f3b-a6fb-b3c69f264d5c","Type":"ContainerStarted","Data":"b670040e55bc906379a950f866efac7b1a1ffe67c4ad2dfb273bbc2a77913a1e"}
Oct 30 05:31:03.202809 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:03.202763    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gszgm" podStartSLOduration=1.092646416 podStartE2EDuration="3.202750469s" totalImagesPullingTime="2.110104053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:03.195810043 +0000 UTC m=+48.922001365" watchObservedRunningTime="2026-10-30 05:31:03.202750469 +0000 UTC m=+48.928941778"
Oct 30 05:31:04.156217 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.156163    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvnfl" event={"ID":"1e3b6130-4545-4c00-ab54-0c40c7a906bb","Type":"ContainerStarted","Data":"53d62b381dc5849bbc53466b8d60aa0a427699399723ae30330bd5a878fb3d83"}
Oct 30 05:31:04.156659 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.156271    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvnfl" event={"ID":"1e3b6130-4545-4c00-ab54-0c40c7a906bb","Type":"ContainerStarted","Data":"2a07735ce62fdb305d699e4d5705b3d5c29a19dcf66f51fa45215b1e10ddc340"}
Oct 30 05:31:04.156659 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.156312    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:04.157688 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.157667    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9665q" event={"ID":"3d0ef442-4160-4ab1-80f1-03b37851a061","Type":"ContainerStarted","Data":"8b72921ca482a09a16aeba62829b81387dbcedc1ec8b63e70c046eecde452481"}
Oct 30 05:31:04.159378 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.159352    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf" event={"ID":"883d9833-be84-411b-998e-cea7379b6263","Type":"ContainerStarted","Data":"5712eb12b4bbc6b8a1adb09356aedfda506c2d18cf0c2ee3afd258de23e86a81"}
Oct 30 05:31:04.159504 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.159403    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf" event={"ID":"883d9833-be84-411b-998e-cea7379b6263","Type":"ContainerStarted","Data":"9358aa67894ecfb50821fafe85a1cec36ed0790c4644a465f6305dd444095013"}
Oct 30 05:31:04.197616 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.197570    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rvnfl" podStartSLOduration=2.210514764 podStartE2EDuration="4.197557011s" totalImagesPullingTime="1.987042247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:04.196958835 +0000 UTC m=+49.923150165" watchObservedRunningTime="2026-10-30 05:31:04.197557011 +0000 UTC m=+49.923748320"
Oct 30 05:31:04.237743 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.237699    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf" podStartSLOduration=1.652272424 podStartE2EDuration="3.237690465s" totalImagesPullingTime="1.585418041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:04.237038399 +0000 UTC m=+49.963229719" watchObservedRunningTime="2026-10-30 05:31:04.237690465 +0000 UTC m=+49.963881774"
Oct 30 05:31:04.276050 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.276010    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9665q" podStartSLOduration=1.353171754 podStartE2EDuration="4.27599738s" totalImagesPullingTime="2.922825626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:04.275424193 +0000 UTC m=+50.001615519" watchObservedRunningTime="2026-10-30 05:31:04.27599738 +0000 UTC m=+50.002188689"
Oct 30 05:31:04.492725 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:04.492651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:05.144420 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.144389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:05.638688 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.638656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:05.802004 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.801974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:05.876068 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.876035    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"]
Oct 30 05:31:05.879070 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.879053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:05.906255 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.906161    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gcmgk\""
Oct 30 05:31:05.906255 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.906162    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 05:31:05.906381 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.906264    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 05:31:05.907618 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.907597    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"]
Oct 30 05:31:05.917637 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.917612    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"]
Oct 30 05:31:05.920515 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.920500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:05.944016 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.943999    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 05:31:05.947740 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.947723    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"]
Oct 30 05:31:05.957244 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.957226    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 05:31:05.957334 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.957318    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 05:31:05.957334 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.957323    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kz82h\""
Oct 30 05:31:05.961593 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.961572    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9h24j"]
Oct 30 05:31:05.964478 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.964462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:05.974215 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.974196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:05.994500 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.994483    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 05:31:05.994665 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.994650    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 05:31:05.994737 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:05.994711    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2znb\""
Oct 30 05:31:06.011660 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.011639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27l79\" (UniqueName: \"kubernetes.io/projected/664945e3-2c17-4cdf-a60d-4c274b0c9006-kube-api-access-27l79\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.011730 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.011689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.011730 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.011706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664945e3-2c17-4cdf-a60d-4c274b0c9006-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.011808 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.011766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.041952 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.041931    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 05:31:06.112710 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.112686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5070dc7e-02b2-450d-a4f8-0f616fbabc99-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.112810 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.112717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-accelerators-collector-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.112810 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.112740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-root\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.112810 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.112754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-wtmp\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.112926 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.112856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.112926 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.112897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664945e3-2c17-4cdf-a60d-4c274b0c9006-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.112986 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.112925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-textfile\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.113017 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.112991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.113017 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.113077 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27l79\" (UniqueName: \"kubernetes.io/projected/664945e3-2c17-4cdf-a60d-4c274b0c9006-kube-api-access-27l79\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.113106 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.113137 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvm9w\" (UniqueName: \"kubernetes.io/projected/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-api-access-lvm9w\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.113257 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.113298 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkt6\" (UniqueName: \"kubernetes.io/projected/1299b210-215f-44c3-9d45-2fda350dc319-kube-api-access-nrkt6\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.113345 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-tls\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.113397 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.113447 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.113501 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-sys\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.113558 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-metrics-client-ca\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.113689 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.113655    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664945e3-2c17-4cdf-a60d-4c274b0c9006-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.115367 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.115349    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.115520 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.115502    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.125763 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.125742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:06.135602 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.135579    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27l79\" (UniqueName: \"kubernetes.io/projected/664945e3-2c17-4cdf-a60d-4c274b0c9006-kube-api-access-27l79\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.193556 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.193488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:06.214717 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvm9w\" (UniqueName: \"kubernetes.io/projected/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-api-access-lvm9w\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.214825 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkt6\" (UniqueName: \"kubernetes.io/projected/1299b210-215f-44c3-9d45-2fda350dc319-kube-api-access-nrkt6\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.214825 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-tls\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.214825 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.214973 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.214973 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-sys\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.214973 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-metrics-client-ca\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.214973 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5070dc7e-02b2-450d-a4f8-0f616fbabc99-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.214973 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-accelerators-collector-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.214973 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-root\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.215288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.214981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-wtmp\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.215288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.215288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-textfile\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.215288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215036    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-sys\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.215288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215065    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-root\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.215288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.215288 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.215742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215681    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-accelerators-collector-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.215742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215698    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-metrics-client-ca\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.215742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215716    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5070dc7e-02b2-450d-a4f8-0f616fbabc99-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.215742 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.215765    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-wtmp\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.216108 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.216086    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-textfile\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.216470 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.216446    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.216568 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.216516    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.217560 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.217539    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.217644 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.217630    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-tls\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.217991 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.217972    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.219251 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.219231    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.241962 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.239307    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkt6\" (UniqueName: \"kubernetes.io/projected/1299b210-215f-44c3-9d45-2fda350dc319-kube-api-access-nrkt6\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.242619 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.242598    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvm9w\" (UniqueName: \"kubernetes.io/projected/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-api-access-lvm9w\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.261681 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.261655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:06.280707 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.280666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:06.291767 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:06.291741    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1299b210_215f_44c3_9d45_2fda350dc319.slice/crio-3bfe50f1f15d3c1b8a2f6a059ce4c86eac2b127060fb2a768164f8fc155b9bd0 WatchSource:0}: Error finding container 3bfe50f1f15d3c1b8a2f6a059ce4c86eac2b127060fb2a768164f8fc155b9bd0: Status 404 returned error can't find the container with id 3bfe50f1f15d3c1b8a2f6a059ce4c86eac2b127060fb2a768164f8fc155b9bd0
Oct 30 05:31:06.400452 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.400422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:06.504401 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.504321    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"]
Oct 30 05:31:06.508271 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:06.508245    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664945e3_2c17_4cdf_a60d_4c274b0c9006.slice/crio-79eeb1e61cb18987fe4ab482fbbb44165beb370a49b6a4130bf40bdd544a8ce4 WatchSource:0}: Error finding container 79eeb1e61cb18987fe4ab482fbbb44165beb370a49b6a4130bf40bdd544a8ce4: Status 404 returned error can't find the container with id 79eeb1e61cb18987fe4ab482fbbb44165beb370a49b6a4130bf40bdd544a8ce4
Oct 30 05:31:06.534811 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.534790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:06.849873 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:06.849787    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"]
Oct 30 05:31:06.853867 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:06.853829    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5070dc7e_02b2_450d_a4f8_0f616fbabc99.slice/crio-1adbf04ed7d16dcac431fceeb3edd585eef53a64a359136b99b0973d38b1d0e4 WatchSource:0}: Error finding container 1adbf04ed7d16dcac431fceeb3edd585eef53a64a359136b99b0973d38b1d0e4: Status 404 returned error can't find the container with id 1adbf04ed7d16dcac431fceeb3edd585eef53a64a359136b99b0973d38b1d0e4
Oct 30 05:31:07.112702 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.112623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:07.169257 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.169153    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9h24j" event={"ID":"1299b210-215f-44c3-9d45-2fda350dc319","Type":"ContainerStarted","Data":"3bfe50f1f15d3c1b8a2f6a059ce4c86eac2b127060fb2a768164f8fc155b9bd0"}
Oct 30 05:31:07.170744 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.170695    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp" event={"ID":"5070dc7e-02b2-450d-a4f8-0f616fbabc99","Type":"ContainerStarted","Data":"1adbf04ed7d16dcac431fceeb3edd585eef53a64a359136b99b0973d38b1d0e4"}
Oct 30 05:31:07.172977 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.172944    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6" event={"ID":"664945e3-2c17-4cdf-a60d-4c274b0c9006","Type":"ContainerStarted","Data":"8a3c32aa7b073417c45c07df0b8c4807813b241d31fd0b6b4fda76292813a873"}
Oct 30 05:31:07.173073 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.172998    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6" event={"ID":"664945e3-2c17-4cdf-a60d-4c274b0c9006","Type":"ContainerStarted","Data":"864e16984800087af37bc34db01aa3dd2dddbd96a3cbd9ae0192ce5c3287194d"}
Oct 30 05:31:07.173073 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.173026    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6" event={"ID":"664945e3-2c17-4cdf-a60d-4c274b0c9006","Type":"ContainerStarted","Data":"79eeb1e61cb18987fe4ab482fbbb44165beb370a49b6a4130bf40bdd544a8ce4"}
Oct 30 05:31:07.253051 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.253011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:07.424376 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.424346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:07.782360 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.782271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:07.981989 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:07.981961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:08.177602 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:08.177566    2579 generic.go:356] "Generic (PLEG): container finished" podID="1299b210-215f-44c3-9d45-2fda350dc319" containerID="d0edc61ff6205706fa8a6456876e2c3cd20ac5f8afbdcad274c74211923515db" exitCode=0
Oct 30 05:31:08.177710 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:08.177645    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9h24j" event={"ID":"1299b210-215f-44c3-9d45-2fda350dc319","Type":"ContainerDied","Data":"d0edc61ff6205706fa8a6456876e2c3cd20ac5f8afbdcad274c74211923515db"}
Oct 30 05:31:08.180053 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:08.180020    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp" event={"ID":"5070dc7e-02b2-450d-a4f8-0f616fbabc99","Type":"ContainerStarted","Data":"abf57cbb3db4d72e0a2191051e34c1fce4b7a1eede8b2c20a6b3f18e90915a04"}
Oct 30 05:31:08.180145 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:08.180078    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp" event={"ID":"5070dc7e-02b2-450d-a4f8-0f616fbabc99","Type":"ContainerStarted","Data":"73bc7061851a698b6fec92fd4a70e94f3e062ab9e93d20ac9ada852dc1a4a184"}
Oct 30 05:31:08.180145 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:08.180105    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp" event={"ID":"5070dc7e-02b2-450d-a4f8-0f616fbabc99","Type":"ContainerStarted","Data":"778185d806814b893f2957765c203bee2702463c94eca959ecd44c3c95f19ed8"}
Oct 30 05:31:08.182437 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:08.182406    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6" event={"ID":"664945e3-2c17-4cdf-a60d-4c274b0c9006","Type":"ContainerStarted","Data":"e8b024348d27ddb54823a80f7987391eaa8d2ff85fcdada0547291fab0009414"}
Oct 30 05:31:08.279469 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:08.278320    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp" podStartSLOduration=2.126157798 podStartE2EDuration="3.278274036s" totalImagesPullingTime="1.152116238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:08.277238729 +0000 UTC m=+54.003430038" watchObservedRunningTime="2026-10-30 05:31:08.278274036 +0000 UTC m=+54.004465357"
Oct 30 05:31:08.323578 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:08.323193    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6" podStartSLOduration=2.366602529 podStartE2EDuration="3.323162466s" totalImagesPullingTime="956.559937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:08.322840991 +0000 UTC m=+54.049032301" watchObservedRunningTime="2026-10-30 05:31:08.323162466 +0000 UTC m=+54.049353775"
Oct 30 05:31:09.176478 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:09.176443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:09.188042 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:09.188010    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9h24j" event={"ID":"1299b210-215f-44c3-9d45-2fda350dc319","Type":"ContainerStarted","Data":"edbaa4b4b0eeda925b331279fc864dec09cd33ff382b85db45ab402ed5bbf06e"}
Oct 30 05:31:09.188217 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:09.188060    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9h24j" event={"ID":"1299b210-215f-44c3-9d45-2fda350dc319","Type":"ContainerStarted","Data":"adf91b817960fab6586273f782076ca455ee1102a9ec5f657705ae2c22d6c220"}
Oct 30 05:31:09.258741 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:09.258695    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9h24j" podStartSLOduration=3.311451552 podStartE2EDuration="4.258683396s" totalImagesPullingTime="862.295052ms" totalInitContainerRuntime="84.936792ms" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:09.246422223 +0000 UTC m=+54.972613532" watchObservedRunningTime="2026-10-30 05:31:09.258683396 +0000 UTC m=+54.984874704"
Oct 30 05:31:09.314480 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:09.314455    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:09.462366 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:09.462296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:09.620836 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:09.620805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:10.177591 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:10.177563    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:10.755836 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:10.755800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:10.967981 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:10.967945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:11.205872 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:11.205827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:12.891041 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:12.891009    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:13.116358 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:13.116326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:13.265823 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:13.265746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:13.466634 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:13.466605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:13.621372 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:13.621285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:14.164541 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:14.164514    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:14.741399 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:14.741373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:14.888520 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:14.888485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:15.025856 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:15.025791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:15.258118 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:15.258086    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:15.404953 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:15.404909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:17.268867 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:17.268831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:17.418280 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:17.418194    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:17.715916 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:17.715882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:17.851735 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:17.851694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:19.615030 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.614997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:19.615030 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.615037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:31:19.628909 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.628880    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 05:31:19.638417 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.638391    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:31:19.640777 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.640761    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 05:31:19.647947 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.647926    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:19.716283 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.716255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:31:19.730107 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.730084    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 05:31:19.735834 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.735813    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w4h88\""
Oct 30 05:31:19.743067 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.743050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:19.750606 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.750589    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 05:31:19.760681 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.760640    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:31:19.818820 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.818783    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-897vl\""
Oct 30 05:31:19.824898 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:19.824878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:31:20.031324 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:20.031288    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c4j6h\""
Oct 30 05:31:20.037450 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:20.037406    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r6plj"]
Oct 30 05:31:20.038370 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:20.038347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:31:20.042117 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:20.042067    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda581d4f8_dc97_47d9_a409_bd391c36e67f.slice/crio-29c13fe87efff85650760063975938ca72c64f7615f22cc0afa736a35d15da46 WatchSource:0}: Error finding container 29c13fe87efff85650760063975938ca72c64f7615f22cc0afa736a35d15da46: Status 404 returned error can't find the container with id 29c13fe87efff85650760063975938ca72c64f7615f22cc0afa736a35d15da46
Oct 30 05:31:20.121309 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:20.121276    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qtv7g"]
Oct 30 05:31:20.126509 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:20.126483    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d421e45_b806_4fcb_b501_62bf9f4079c0.slice/crio-5a605957857e77d7c9ca1148e326e72d51619efc12a18c6ed2a134841094ab14 WatchSource:0}: Error finding container 5a605957857e77d7c9ca1148e326e72d51619efc12a18c6ed2a134841094ab14: Status 404 returned error can't find the container with id 5a605957857e77d7c9ca1148e326e72d51619efc12a18c6ed2a134841094ab14
Oct 30 05:31:20.226523 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:20.226490    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qtv7g" event={"ID":"3d421e45-b806-4fcb-b501-62bf9f4079c0","Type":"ContainerStarted","Data":"5a605957857e77d7c9ca1148e326e72d51619efc12a18c6ed2a134841094ab14"}
Oct 30 05:31:20.227748 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:20.227722    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r6plj" event={"ID":"a581d4f8-dc97-47d9-a409-bd391c36e67f","Type":"ContainerStarted","Data":"29c13fe87efff85650760063975938ca72c64f7615f22cc0afa736a35d15da46"}
Oct 30 05:31:20.333087 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:20.333054    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njsfm"]
Oct 30 05:31:20.336724 ip-10-0-130-194 kubenswrapper[2579]: W1030 05:31:20.336692    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90d7ebe_df62_49e8_97e7_103ba14cd129.slice/crio-7f7b59b23ad26a44fd3b90f1fad8f1c7748d69162bd0ecae3ee6b33c082be49c WatchSource:0}: Error finding container 7f7b59b23ad26a44fd3b90f1fad8f1c7748d69162bd0ecae3ee6b33c082be49c: Status 404 returned error can't find the container with id 7f7b59b23ad26a44fd3b90f1fad8f1c7748d69162bd0ecae3ee6b33c082be49c
Oct 30 05:31:20.962233 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:20.962173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:21.165323 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:21.165293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:21.234645 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:21.234616    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njsfm" event={"ID":"b90d7ebe-df62-49e8-97e7-103ba14cd129","Type":"ContainerStarted","Data":"7f7b59b23ad26a44fd3b90f1fad8f1c7748d69162bd0ecae3ee6b33c082be49c"}
Oct 30 05:31:21.305038 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:21.305007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:22.171812 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:22.171773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:22.241380 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:22.241076    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qtv7g" event={"ID":"3d421e45-b806-4fcb-b501-62bf9f4079c0","Type":"ContainerStarted","Data":"4b7469dd1e8544fc2d482b892dd8263ae82e43cc872fa33e7c307fc71b3c0004"}
Oct 30 05:31:22.241380 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:22.241200    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qtv7g" event={"ID":"3d421e45-b806-4fcb-b501-62bf9f4079c0","Type":"ContainerStarted","Data":"095eca260ae8be012a20c5423a9a63ce0fb562d412596c4b6e6933df74e96621"}
Oct 30 05:31:22.281529 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:22.281473    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qtv7g" podStartSLOduration=67.251567475 podStartE2EDuration="1m8.281456943s" totalImagesPullingTime="1.029889468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:22.281315976 +0000 UTC m=+68.007507285" watchObservedRunningTime="2026-10-30 05:31:22.281456943 +0000 UTC m=+68.007648254"
Oct 30 05:31:22.311258 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:22.311221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:23.395492 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:23.395389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:23.539089 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:23.539060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:24.251099 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:24.251060    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njsfm" event={"ID":"b90d7ebe-df62-49e8-97e7-103ba14cd129","Type":"ContainerStarted","Data":"fb90295f71a1e9bfdf079d3e39e185ba0388eba685ae49cf5e470bfd32bcde1f"}
Oct 30 05:31:24.251344 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:24.251321    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:31:24.292070 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:24.290670    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njsfm" podStartSLOduration=66.089893785 podStartE2EDuration="1m9.290652538s" totalImagesPullingTime="3.200758753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:24.290068891 +0000 UTC m=+70.016260203" watchObservedRunningTime="2026-10-30 05:31:24.290652538 +0000 UTC m=+70.016843848"
Oct 30 05:31:24.686088 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:24.686057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:24.843017 ip-10-0-130-194 kubenswrapper[2579]: I1030 05:31:24.842979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:25.100813 ip-10-0-130-194 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 05:31:25.117556 ip-10-0-130-194 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 05:31:25.117728 ip-10-0-130-194 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 05:31:25.117778 ip-10-0-130-194 systemd[1]: kubelet.service: Consumed 3.846s CPU time.
Oct 30 05:31:25.129923 ip-10-0-130-194 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 05:31:25.198423 ip-10-0-130-194 kubenswrapper[5250]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 05:31:25.198423 ip-10-0-130-194 kubenswrapper[5250]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 05:31:25.198423 ip-10-0-130-194 kubenswrapper[5250]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 05:31:25.198423 ip-10-0-130-194 kubenswrapper[5250]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 05:31:25.200486 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200404    5250 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:31:25.200486 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200486    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:31:25.200486 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200489    5250 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200494    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200497    5250 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200499    5250 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200502    5250 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200505    5250 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200507    5250 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200510    5250 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200512    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200515    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200520    5250 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200524    5250 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200527    5250 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200540    5250 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200543    5250 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200546    5250 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200548    5250 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200551    5250 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200553    5250 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200556    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:31:25.200599 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200559    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200561    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200564    5250 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200567    5250 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200570    5250 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200573    5250 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200575    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200578    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200580    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200583    5250 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200586    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200589    5250 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200592    5250 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200594    5250 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200597    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200599    5250 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200602    5250 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200605    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200607    5250 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200610    5250 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:31:25.201069 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200613    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200615    5250 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200618    5250 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200620    5250 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200623    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200626    5250 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200629    5250 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200631    5250 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200634    5250 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200637    5250 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200639    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200642    5250 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200644    5250 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200647    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200650    5250 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200653    5250 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200655    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200658    5250 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200660    5250 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200663    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:31:25.201642 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200665    5250 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200668    5250 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200671    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200674    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200677    5250 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200679    5250 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200684    5250 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200687    5250 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200690    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200693    5250 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200696    5250 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200699    5250 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200701    5250 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200704    5250 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200706    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200709    5250 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200711    5250 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200714    5250 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200717    5250 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:31:25.202116 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200719    5250 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200722    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200724    5250 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200727    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200729    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200732    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200736    5250 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200738    5250 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200741    5250 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200743    5250 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200746    5250 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200748    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200751    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200754    5250 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200757    5250 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200759    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200762    5250 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200765    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200768    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:31:25.202612 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200770    5250 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200772    5250 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200775    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200778    5250 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200781    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.200784    5250 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201821    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201830    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201833    5250 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201836    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201839    5250 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201842    5250 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201845    5250 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201848    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201851    5250 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201854    5250 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201857    5250 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201859    5250 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201862    5250 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:31:25.203099 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201865    5250 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201867    5250 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201870    5250 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201872    5250 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201875    5250 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201877    5250 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201880    5250 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201882    5250 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201886    5250 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201889    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201891    5250 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201894    5250 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201898    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201900    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201903    5250 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201906    5250 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201909    5250 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201911    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201914    5250 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201917    5250 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:31:25.203579 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201919    5250 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201922    5250 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201925    5250 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201927    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201930    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201933    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201936    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201938    5250 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201941    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201944    5250 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201946    5250 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201949    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201951    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201954    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201957    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201959    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201962    5250 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201964    5250 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201967    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:31:25.204062 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201969    5250 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201972    5250 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201974    5250 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201977    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201980    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201983    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201991    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201994    5250 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201996    5250 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.201999    5250 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202002    5250 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202005    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202007    5250 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202010    5250 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202013    5250 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202015    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202018    5250 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202020    5250 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202023    5250 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202026    5250 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:31:25.204574 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202028    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202031    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202033    5250 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202036    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202038    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202042    5250 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202047    5250 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202051    5250 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202054    5250 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202056    5250 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202059    5250 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202062    5250 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202064    5250 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202067    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202069    5250 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202072    5250 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202074    5250 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202077    5250 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202079    5250 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:31:25.205056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202089    5250 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202092    5250 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202094    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202097    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202100    5250 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202102    5250 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202105    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202108    5250 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202110    5250 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202113    5250 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202116    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202119    5250 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202121    5250 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202124    5250 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202127    5250 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202627    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202633    5250 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202636    5250 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202639    5250 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202642    5250 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:31:25.205557 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202644    5250 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202647    5250 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202649    5250 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202652    5250 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202654    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202657    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202660    5250 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202663    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202666    5250 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202668    5250 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202671    5250 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202673    5250 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202677    5250 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202680    5250 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202682    5250 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202685    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202688    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202690    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202693    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:31:25.206037 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202696    5250 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202698    5250 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202701    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202703    5250 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202706    5250 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202708    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202711    5250 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202713    5250 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202716    5250 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202719    5250 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202721    5250 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202723    5250 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202727    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202730    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202732    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202734    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202737    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202739    5250 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202742    5250 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:31:25.206507 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202745    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202747    5250 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202750    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202754    5250 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202757    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202760    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202763    5250 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202769    5250 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202772    5250 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202775    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202778    5250 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202780    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202783    5250 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202786    5250 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202788    5250 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202791    5250 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202793    5250 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202796    5250 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202799    5250 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:31:25.207298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202801    5250 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202804    5250 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202806    5250 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202809    5250 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202811    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202814    5250 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202816    5250 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202819    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202821    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202824    5250 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202826    5250 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202829    5250 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202831    5250 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202834    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202837    5250 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202839    5250 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202842    5250 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202844    5250 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202847    5250 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202849    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:31:25.208090 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202853    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202856    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202859    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202861    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202864    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202866    5250 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202869    5250 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202871    5250 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202873    5250 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202876    5250 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202879    5250 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202883    5250 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202887    5250 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202890    5250 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202893    5250 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202895    5250 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202900    5250 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202903    5250 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202906    5250 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202909    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:31:25.208889 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202911    5250 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202914    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202917    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.202919    5250 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203013    5250 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203020    5250 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203029    5250 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203035    5250 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203041    5250 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203046    5250 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203052    5250 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203056    5250 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203061    5250 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203064    5250 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203068    5250 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203072    5250 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203075    5250 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203078    5250 flags.go:64] FLAG: --cgroup-root=""
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203081    5250 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203084    5250 flags.go:64] FLAG: --client-ca-file=""
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203087    5250 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203090    5250 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203095    5250 flags.go:64] FLAG: --cluster-domain=""
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203098    5250 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 05:31:25.209607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203101    5250 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203104    5250 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203108    5250 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203111    5250 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203116    5250 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203119    5250 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203122    5250 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203125    5250 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203128    5250 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203131    5250 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203134    5250 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203137    5250 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203141    5250 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203144    5250 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203147    5250 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203150    5250 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203153    5250 flags.go:64] FLAG: --enable-server="true"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203156    5250 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203160    5250 flags.go:64] FLAG: --event-burst="100"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203163    5250 flags.go:64] FLAG: --event-qps="50"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203167    5250 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203170    5250 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203186    5250 flags.go:64] FLAG: --eviction-hard=""
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203191    5250 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 05:31:25.210212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203194    5250 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203197    5250 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203200    5250 flags.go:64] FLAG: --eviction-soft=""
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203203    5250 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203206    5250 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203209    5250 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203212    5250 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203214    5250 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203217    5250 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203220    5250 flags.go:64] FLAG: --feature-gates=""
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203224    5250 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203227    5250 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203232    5250 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203235    5250 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203238    5250 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203241    5250 flags.go:64] FLAG: --help="false"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203244    5250 flags.go:64] FLAG: --hostname-override="ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203247    5250 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203250    5250 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203253    5250 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203256    5250 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203259    5250 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203262    5250 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 05:31:25.210869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203265    5250 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203268    5250 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203271    5250 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203274    5250 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203277    5250 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203281    5250 flags.go:64] FLAG: --kube-reserved=""
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203284    5250 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203287    5250 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203292    5250 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203297    5250 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203301    5250 flags.go:64] FLAG: --lock-file=""
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203306    5250 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203310    5250 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203313    5250 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203318    5250 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203321    5250 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203324    5250 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203327    5250 flags.go:64] FLAG: --logging-format="text"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203330    5250 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203333    5250 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203336    5250 flags.go:64] FLAG: --manifest-url=""
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203340    5250 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203344    5250 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203347    5250 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203352    5250 flags.go:64] FLAG: --max-pods="110"
Oct 30 05:31:25.211503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203355    5250 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203358    5250 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203361    5250 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203364    5250 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203367    5250 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203370    5250 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203372    5250 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203380    5250 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203383    5250 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203386    5250 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203389    5250 flags.go:64] FLAG: --pod-cidr=""
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203392    5250 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203397    5250 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203400    5250 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203402    5250 flags.go:64] FLAG: --port="10250"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203405    5250 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203408    5250 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08c24ae267f12a1dc"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203412    5250 flags.go:64] FLAG: --qos-reserved=""
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203416    5250 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203419    5250 flags.go:64] FLAG: --register-node="true"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203421    5250 flags.go:64] FLAG: --register-with-taints=""
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203425    5250 flags.go:64] FLAG: --registry-burst="10"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203428    5250 flags.go:64] FLAG: --registry-qps="5"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203431    5250 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203434    5250 flags.go:64] FLAG: --reserved-memory=""
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203438    5250 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 05:31:25.212120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203441    5250 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203444    5250 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203448    5250 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203451    5250 flags.go:64] FLAG: --runonce="false"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203453    5250 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203456    5250 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203459    5250 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203462    5250 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203465    5250 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203468    5250 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203471    5250 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203474    5250 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203477    5250 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203480    5250 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203483    5250 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203486    5250 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203489    5250 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203491    5250 flags.go:64] FLAG: --system-cgroups=""
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203495    5250 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203500    5250 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203503    5250 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203506    5250 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203510    5250 flags.go:64] FLAG: --tls-min-version=""
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203513    5250 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203516    5250 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 05:31:25.212835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203520    5250 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203523    5250 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203526    5250 flags.go:64] FLAG: --v="2"
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203530    5250 flags.go:64] FLAG: --version="false"
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203534    5250 flags.go:64] FLAG: --vmodule=""
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203538    5250 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203541    5250 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203681    5250 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203685    5250 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203690    5250 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203693    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203695    5250 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203698    5250 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203701    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203703    5250 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203706    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203709    5250 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203711    5250 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203713    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203716    5250 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203718    5250 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:31:25.213570 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203721    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203723    5250 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203726    5250 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203729    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203732    5250 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203735    5250 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203737    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203740    5250 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203742    5250 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203745    5250 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203747    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203750    5250 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203753    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203756    5250 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203758    5250 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203761    5250 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203764    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203766    5250 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203769    5250 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203771    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:31:25.214075 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203775    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203777    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203780    5250 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203782    5250 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203785    5250 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203787    5250 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203790    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203794    5250 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203798    5250 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203801    5250 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203804    5250 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203807    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203809    5250 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203812    5250 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203815    5250 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203817    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203820    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203822    5250 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203825    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:31:25.214799 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203827    5250 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203830    5250 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203832    5250 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203835    5250 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203837    5250 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203840    5250 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203843    5250 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203845    5250 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203848    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203851    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203853    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203856    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203858    5250 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203862    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203865    5250 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203867    5250 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203870    5250 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203873    5250 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203875    5250 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203878    5250 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:31:25.215365 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203880    5250 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203883    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203885    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203888    5250 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203890    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203894    5250 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203898    5250 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203901    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203904    5250 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203906    5250 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203909    5250 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203911    5250 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203914    5250 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203916    5250 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203919    5250 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203921    5250 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203924    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203926    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203929    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:31:25.215971 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203932    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203934    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203937    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203940    5250 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203942    5250 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203945    5250 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203949    5250 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203951    5250 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203954    5250 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203956    5250 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203959    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203961    5250 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203964    5250 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.203967    5250 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.203992    5250 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204028    5250 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:31:25.216919 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204032    5250 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204035    5250 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204038    5250 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204041    5250 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204043    5250 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204046    5250 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204048    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204051    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204054    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204057    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204059    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204062    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204064    5250 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204067    5250 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204070    5250 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204073    5250 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204076    5250 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204078    5250 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204083    5250 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204085    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:31:25.217627 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204088    5250 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204090    5250 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204093    5250 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204097    5250 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204100    5250 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204103    5250 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204105    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204108    5250 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204110    5250 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204113    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204115    5250 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204118    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204120    5250 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204123    5250 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204126    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204128    5250 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204131    5250 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204133    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204136    5250 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:31:25.218211 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204138    5250 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204141    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204143    5250 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204146    5250 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204148    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204151    5250 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204153    5250 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204156    5250 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204158    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204161    5250 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204165    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204169    5250 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204172    5250 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204190    5250 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204192    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204195    5250 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204199    5250 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204202    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204205    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204207    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:31:25.218751 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204210    5250 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204213    5250 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204215    5250 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204218    5250 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204221    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204225    5250 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204228    5250 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204231    5250 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204234    5250 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204237    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204240    5250 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204243    5250 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204245    5250 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204248    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204250    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204253    5250 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204256    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204258    5250 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204261    5250 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:31:25.219326 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204263    5250 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204266    5250 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204268    5250 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204271    5250 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204275    5250 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204277    5250 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204280    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204283    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204286    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204290    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204292    5250 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204296    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204298    5250 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204301    5250 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204303    5250 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204306    5250 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204309    5250 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204311    5250 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204314    5250 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:31:25.219790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204316    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204319    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204321    5250 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204324    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204328    5250 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204332    5250 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204336    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.204348    5250 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.209916    5250 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.209938    5250 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.209956    5250 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.209962    5250 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210034    5250 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210039    5250 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210043    5250 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210046    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210050    5250 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210052    5250 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210055    5250 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:31:25.220368 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210058    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210061    5250 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210063    5250 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210066    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210068    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210071    5250 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210074    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210076    5250 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210079    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210082    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210084    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210087    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210090    5250 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210093    5250 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210096    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210098    5250 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210101    5250 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210104    5250 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210106    5250 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:31:25.220824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210109    5250 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210115    5250 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210118    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210121    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210124    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210127    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210129    5250 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210132    5250 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210134    5250 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210137    5250 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210139    5250 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210142    5250 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210145    5250 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210147    5250 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210150    5250 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210152    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210155    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210157    5250 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210160    5250 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210162    5250 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:31:25.221311 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210166    5250 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210168    5250 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210171    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210194    5250 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210197    5250 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210200    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210203    5250 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210205    5250 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210209    5250 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210212    5250 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210215    5250 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210217    5250 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210220    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210224    5250 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210227    5250 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210230    5250 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210232    5250 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210235    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210238    5250 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210240    5250 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:31:25.221828 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210243    5250 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210245    5250 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210250    5250 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210253    5250 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210256    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210259    5250 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210262    5250 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210266    5250 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210269    5250 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210272    5250 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210275    5250 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210277    5250 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210280    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210283    5250 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210286    5250 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210289    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210291    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210294    5250 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210296    5250 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:31:25.222321 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210299    5250 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210302    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210305    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210307    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210310    5250 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210313    5250 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210315    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210318    5250 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210320    5250 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210323    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210325    5250 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210328    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210330    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210333    5250 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210335    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210338    5250 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210340    5250 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210343    5250 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210345    5250 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210348    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:31:25.222790 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210350    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.210378    5250 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210515    5250 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210519    5250 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210522    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210525    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210528    5250 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210531    5250 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210533    5250 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210536    5250 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210538    5250 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210541    5250 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210544    5250 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210547    5250 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210550    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210552    5250 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:31:25.223298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210556    5250 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210560    5250 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210563    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210566    5250 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210570    5250 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210573    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210576    5250 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210580    5250 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210582    5250 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210585    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210588    5250 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210590    5250 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210593    5250 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210596    5250 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210598    5250 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210601    5250 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210604    5250 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210606    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210609    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:31:25.223689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210611    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210614    5250 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210617    5250 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210619    5250 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210622    5250 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210625    5250 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210628    5250 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210631    5250 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210633    5250 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210636    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210639    5250 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210642    5250 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210644    5250 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210647    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210649    5250 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210652    5250 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210654    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210657    5250 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210660    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:31:25.224154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210662    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210665    5250 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210667    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210670    5250 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210672    5250 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210675    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210678    5250 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210680    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210683    5250 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210685    5250 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210688    5250 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210690    5250 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210693    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210695    5250 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210698    5250 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210700    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210703    5250 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210706    5250 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210708    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:31:25.224893 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210712    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210714    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210717    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210719    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210725    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210728    5250 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210730    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210733    5250 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210737    5250 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210741    5250 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210746    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210749    5250 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210753    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210757    5250 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210760    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210763    5250 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210766    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210768    5250 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210771    5250 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210774    5250 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:31:25.225459 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210777    5250 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210780    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210782    5250 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210785    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210788    5250 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210790    5250 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210793    5250 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210796    5250 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210798    5250 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210801    5250 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210803    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210806    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210810    5250 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210815    5250 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210819    5250 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.210845    5250 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 05:31:25.226070 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210882    5250 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210886    5250 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210889    5250 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210892    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210895    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210897    5250 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210900    5250 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210903    5250 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210905    5250 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210908    5250 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210910    5250 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210913    5250 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210915    5250 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210918    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210920    5250 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210924    5250 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210929    5250 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210932    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210936    5250 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 05:31:25.226672 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210940    5250 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210944    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210947    5250 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210949    5250 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210952    5250 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210954    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210957    5250 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210960    5250 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210962    5250 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210965    5250 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210967    5250 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210970    5250 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210973    5250 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210978    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210982    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210986    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210990    5250 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210994    5250 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.210998    5250 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211001    5250 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 05:31:25.227335 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211005    5250 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211009    5250 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211013    5250 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211017    5250 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211021    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211025    5250 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211029    5250 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211033    5250 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211038    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211042    5250 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211046    5250 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211050    5250 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211054    5250 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211058    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211062    5250 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211066    5250 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211071    5250 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211075    5250 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211080    5250 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 05:31:25.227977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211084    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211089    5250 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211092    5250 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211095    5250 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211097    5250 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211102    5250 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211107    5250 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211110    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211113    5250 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211115    5250 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211118    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211120    5250 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211123    5250 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211125    5250 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211128    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211131    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211135    5250 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211138    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211140    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 05:31:25.228531 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211143    5250 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211146    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211148    5250 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211151    5250 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211153    5250 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211156    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211159    5250 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211161    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211164    5250 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211166    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211169    5250 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211171    5250 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211191    5250 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211195    5250 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211199    5250 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211202    5250 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211205    5250 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211207    5250 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211210    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211216    5250 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 05:31:25.228991 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211218    5250 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211221    5250 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211224    5250 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211226    5250 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211229    5250 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211231    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211234    5250 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211237    5250 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:31:25.211240    5250 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.211312    5250 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.212348    5250 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.212419    5250 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.212618    5250 server.go:1014] "Starting client certificate rotation"
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.212791    5250 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.212842    5250 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:25:14 +0000 UTC" deadline="2028-06-17 18:45:27.70942639 +0000 UTC"
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.212875    5250 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14317h14m2.496553455s"
Oct 30 05:31:25.229566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.213151    5250 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 05:31:25.229958 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.214418    5250 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 05:31:25.229958 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.215676    5250 log.go:25] "Validated CRI v1 runtime API"
Oct 30 05:31:25.229958 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.217762    5250 log.go:25] "Validated CRI v1 image API"
Oct 30 05:31:25.229958 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.218519    5250 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 05:31:25.229958 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.220366    5250 fs.go:135] Filesystem UUIDs: map[139e0d9f-f0c0-41fd-9c44-a8fc2b553f61:/dev/nvme0n1p4 391513df-e5c0-4a32-b66e-64f1870c5773:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 05:31:25.230119 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.220389    5250 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fae5a73bc0bec1c8fe0868046ec1d65af718f76d314d329413413eff7ff5d17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fae5a73bc0bec1c8fe0868046ec1d65af718f76d314d329413413eff7ff5d17/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ff467ebb2cfce1d0fa7a2aa50f84e935dd4f08a8f877b7dbb746050c6e1382a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ff467ebb2cfce1d0fa7a2aa50f84e935dd4f08a8f877b7dbb746050c6e1382a/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1adbf04ed7d16dcac431fceeb3edd585eef53a64a359136b99b0973d38b1d0e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1adbf04ed7d16dcac431fceeb3edd585eef53a64a359136b99b0973d38b1d0e4/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e9dbf9926bdb9e9741a66e8cbe7e0e2fb0438702fc2a99ec9db630a08183943/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e9dbf9926bdb9e9741a66e8cbe7e0e2fb0438702fc2a99ec9db630a08183943/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29c13fe87efff85650760063975938ca72c64f7615f22cc0afa736a35d15da46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29c13fe87efff85650760063975938ca72c64f7615f22cc0afa736a35d15da46/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bfe50f1f15d3c1b8a2f6a059ce4c86eac2b127060fb2a768164f8fc155b9bd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bfe50f1f15d3c1b8a2f6a059ce4c86eac2b127060fb2a768164f8fc155b9bd0/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c44b9e34b90be5c951572b3193574cf979d3b7c87cf6b201c13305f8372fc09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c44b9e34b90be5c951572b3193574cf979d3b7c87cf6b201c13305f8372fc09/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55ddd6fa2f4f279b34fc1061da2f63bf24e4530c320964a81c7a91d18636d1ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55ddd6fa2f4f279b34fc1061da2f63bf24e4530c320964a81c7a91d18636d1ce/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a605957857e77d7c9ca1148e326e72d51619efc12a18c6ed2a134841094ab14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a605957857e77d7c9ca1148e326e72d51619efc12a18c6ed2a134841094ab14/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b7c849b7b57f9a3ed10cfba9d508c1dd1982e5d378a7a467f057f695db60d52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b7c849b7b57f9a3ed10cfba9d508c1dd1982e5d378a7a467f057f695db60d52/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62ed862a2a771f7c861fd0b52d87b5fcb9805e8259b4a1dd61b34fcd9d17255a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62ed862a2a771f7c861fd0b52d87b5fcb9805e8259b4a1dd61b34fcd9d17255a/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e6bf8f2095624f2453cd1741d8801b5838db0d6f7ae237041956acbb352ce0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e6bf8f2095624f2453cd1741d8801b5838db0d6f7ae237041956acbb352ce0d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79eeb1e61cb18987fe4ab482fbbb44165beb370a49b6a4130bf40bdd544a8ce4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79eeb1e61cb18987fe4ab482fbbb44165beb370a49b6a4130bf40bdd544a8ce4/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f7b59b23ad26a44fd3b90f1fad8f1c7748d69162bd0ecae3ee6b33c082be49c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f7b59b23ad26a44fd3b90f1fad8f1c7748d69162bd0ecae3ee6b33c082be49c/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86e0d6284c01e3ba8246caab4a79288509c1e8f3e698088c2891223c178048ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86e0d6284c01e3ba8246caab4a79288509c1e8f3e698088c2891223c178048ce/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdd2baec5fd002bdb88c1a9db7ecd659bbd78a9a8accd057bd6eab1cd4203eb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdd2baec5fd002bdb88c1a9db7ecd659bbd78a9a8accd057bd6eab1cd4203eb0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf928504a563cea3e752b24a491b5a8ca88db0c13e126d1fe38998d0c6c1e6d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf928504a563cea3e752b24a491b5a8ca88db0c13e126d1fe38998d0c6c1e6d5/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0b6012dc0cee2e0173d1c27ac78d3970fb85701d1e9cf7074195dd710a26d9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0b6012dc0cee2e0173d1c27ac78d3970fb85701d1e9cf7074195dd710a26d9c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1710ec37388ba46b5e004e7cea4f0b74eb72c88ae2b795deae975ac891d63a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1710ec37388ba46b5e004e7cea4f0b74eb72c88ae2b795deae975ac891d63a7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe52fdd1efdf4085b572c56f940fd14235d8cb1be3a3e84459df6b6c164ad45f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe52fdd1efdf4085b572c56f940fd14235d8cb1be3a3e84459df6b6c164ad45f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/087e82f9-024f-4bcd-97cf-8b7f928b719a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/087e82f9-024f-4bcd-97cf-8b7f928b719a/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~projected/kube-api-access-nrkt6:{mountpoint:/var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~projected/kube-api-access-nrkt6 major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/180d488f-b7df-4f3b-a6fb-b3c69f264d5c/volumes/kubernetes.io~projected/kube-api-access-vlh8n:{mountpoint:/var/lib/kubelet/pods/180d488f-b7df-4f3b-a6fb-b3c69f264d5c/volumes/kubernetes.io~projected/kube-api-access-vlh8n major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/180d488f-b7df-4f3b-a6fb-b3c69f264d5c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/180d488f-b7df-4f3b-a6fb-b3c69f264d5c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e3b6130-4545-4c00-ab54-0c40c7a906bb/volumes/kubernetes.io~projected/kube-api-access-b7bb4:{mountpoint:/var/lib/kubelet/pods/1e3b6130-4545-4c00-ab54-0c40c7a906bb/volumes/kubernetes.io~projected/kube-api-access-b7bb4 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e3b6130-4545-4c00-ab54-0c40c7a906bb/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1e3b6130-4545-4c00-ab54-0c40c7a906bb/volumes/kubernetes.io~secret/metrics-tls major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d0ef442-4160-4ab1-80f1-03b37851a061/volumes/kubernetes.io~projected/kube-api-access-b2r6d:{mountpoint:/var/lib/kubelet/pods/3d0ef442-4160-4ab1-80f1-03b37851a061/volumes/kubernetes.io~projected/kube-api-access-b2r6d major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d0ef442-4160-4ab1-80f1-03b37851a061/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3d0ef442-4160-4ab1-80f1-03b37851a061/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d421e45-b806-4fcb-b501-62bf9f4079c0/volumes/kubernetes.io~projected/kube-api-access-cdbbp:{mountpoint:/var/lib/kubelet/pods/3d421e45-b806-4fcb-b501-62bf9f4079c0/volumes/kubernetes.io~projected/kube-api-access-cdbbp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d421e45-b806-4fcb-b501-62bf9f4079c0/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3d421e45-b806-4fcb-b501-62bf9f4079c0/volumes/kubernetes.io~secret/metrics-certs major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdbc4e4-f579-4d7b-abc3-295b81e4f468/volumes/kubernetes.io~projected/kube-api-access-2nmnv:{mountpoint:/var/lib/kubelet/pods/4bdbc4e4-f579-4d7b-abc3-295b81e4f468/volumes/kubernetes.io~projected/kube-api-access-2nmnv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~projected/kube-api-access-lvm9w:{mountpoint:/var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~projected/kube-api-access-lvm9w major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volumes/kubernetes.io~projected/kube-api-access-f8chr:{mountpoint:/var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volumes/kubernetes.io~projected/kube-api-access-f8chr major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64b35bb9-7f5b-4009-a6cf-15acaf83b02f/volumes/kubernetes.io~projected/kube-api-access-wx2c4:{mountpoint:/var/lib/kubelet/pods/64b35bb9-7f5b-4009-a6cf-15acaf83b02f/volumes/kubernetes.io~projected/kube-api-access-wx2c4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~projected/kube-api-access-27l79:{mountpoint:/var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~projected/kube-api-access-27l79 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81290ea5-5539-4ab9-9747-673e98c728d4/volumes/kubernetes.io~projected/kube-api-access-lwq48:{mountpoint:/var/lib/kubelet/pods/81290ea5-5539-4ab9-9747-673e98c728d4/volumes/kubernetes.io~projected/kube-api-access-lwq48 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/840883aa-27bd-49c8-a676-1d8336ec90c2/volumes/kubernetes.io~projected/kube-api-access-c6j95:{mountpoint:/var/lib/kubelet/pods/840883aa-27bd-49c8-a676-1d8336ec90c2/volumes/kubernetes.io~projected/kube-api-access-c6j95 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~projected/kube-api-access-b7c8g:{mountpoint:/var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~projected/kube-api-access-b7c8g major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a581d4f8-dc97-47d9-a409-bd391c36e67f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a581d4f8-dc97-47d9-a409-bd391c36e67f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab696c88-f337-4fd8-a9d7-be141c4c421c/volumes/kubernetes.io~projected/kube-api-access-xw7x2:{mountpoint:/var/lib/kubelet/pods/ab696c88-f337-4fd8-a9d7-be141c4c421c/volumes/kubernetes.io~projected/kube-api-access-xw7x2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adb28f6d-9926-4b6a-b37a-149ae4785e1a/volumes/kubernetes.io~projected/kube-api-access-kf8kh:{mountpoint:/var/lib/kubelet/pods/adb28f6d-9926-4b6a-b37a-149ae4785e1a/volumes/kubernetes.io~projected/kube-api-access-kf8kh major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b90d7ebe-df62-49e8-97e7-103ba14cd129/volumes/kubernetes.io~projected/kube-api-access-4q7vq:{mountpoint:/var/lib/kubelet/pods/b90d7ebe-df62-49e8-97e7-103ba14cd129/volumes/kubernetes.io~projected/kube-api-access-4q7vq major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~projected/kube-api-access-nkskp:{mountpoint:/var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~projected/kube-api-access-nkskp major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/97613b67fe5a342c043c016436537d20f798964cb64e936937c503eeb0bc2a2a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/997a170d4353903f3ccc14b7c6eef8e4836e5607fbb47a1cd9e92be97400be22/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b6f9719707a638d06cf2872c142aa542639afe98c24e63b8ff23c80e9cc88797/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/499851cc1c3a819b873e660c760cc3f2825ccad0aea01226dd0af7d355da0da0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/388b1ec7211145bad8b8018de8f3de79a807102653027f4ee831f12e781368aa/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/da053faa66f01585437e3c408e3f494500dd596d8ef6f154784d37316d2df17d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c82a92edb7841548a048b9f45095b404c74660762148a8d0641303ccd660bc15/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d65634193e057ebc040d9396d13951e893ec6a0e1777196f9c7a62449ecf05f8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6f9a689c89315a944cbd1664c7d93050e7817e2cc9dcd904028388e300244d8a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/125bc83a2c40937405c58fa3d6eb88785b7e6021196bf6c5d16f059850dfa71d/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/4ff7744761b3c80d11019101b8748080faa0726d9d838381b480b95a4692fbbf/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/a39586ef534828c2a5c594c152b64db4b216d7c2a1543c2c38171982fe81cbd7/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/76009a10385a4b5fa1b064fc9e060f905b2c839d492e44fc3157a56bf8d133d8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e337afb39398b8fcae9626d16d3052391431705f03c36120b2175e06cb9a0e8a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/ea47149afa467db341bf8a1083e18dfbb3cc98cd49f5c26a5319d27a3634d165/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/46788f9bad141c38471d96cd5df8d36631d20f6e3473f4e2373cd4b43ed14605/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/60977c10b0a13225951e8472e3d517800bbe535ad5f6b3fec0c91ff830358dde/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6ffc5317d7cef7eabd1772e1c05af7f5a441c8e8d0ba63caa98ea92bfe96b1b8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4adf1876492b92ba11f312e561c35baa3dbd2f14dc4e9da14fedd8f83ce39f8a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/31381d4b807a36df80292906b5e680989a7557857bd640d3b21e7f24ed5a7fa1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/11fd2fbdab3684e4265d701cea30c00724a036ea9ff12ea98dff075f82430e33/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/d5d81f56e2d1da71a38f30cd52a781dbaca5df65ed849d041f7d169476a2d98f/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7fdec1bf3ba6225fa05a7e97185ffcb4f784f19ab27b0c6858d0dfed6a2868e3/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e4e973b77234f4bb47820130fed62452b0a9ee24c7192df7e8808638fb9f1593/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/950e982c9a409a0e90115a4ab8355c892811ff42a3830a88e8e748d7e36c3c0c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/24c769ea6039c9f2eabc07567ae36ec1f5ae59d06d3dc81d5e05198efbe2b3cf/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/8f7f3bafaf578b92096c91ccb5754826abc28693259a40b381c2821bcf75711c/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/ecc61de7206b914efdbd80f8f8404b2cc4ff2891a9ffc2bbce6828030e2b410d/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/606ca8db5ca05590c5b77e40d58447a65ef1fbbc59c66eb61acb08ba78df0a78/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/9ed17471f8bef9333de42df720ad6cc0c816356485e1a72df196c562406d147a/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ea423aa8ca611d8df6a458b09b3def189f0b534e2414abb99b80b4b3dd600772/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/f6f0b554b3f60be097899c00839df3f6c7bd19186791a9b68f44fc1264502ebb/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/72be9cb9c634f0d027b123c8961b9fd7ec0a60726d56fea2b058cbda35f83e89/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/333aa37f5b6448749aaa4900dfd346131273fc79e8edf0bec61f1683cbbf218f/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/4abf17878ef96a9ea7cdd787cbe9d59144807336ee847033804598d6d2a7b41e/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/ead9de736fc02f8aabcc549f68467b8b1e2f6498779c9ea2b946c58d9405392b/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/9c7818036220854d43778145b2f2f2ff908b64bb3544eec24a87ed939e8b3e3d/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/19db328c4c06f0c0fa0b89e56d010bc0091e2c7bd8c6c548df5a67aeccc8cc7f/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/8424efec8938f13565ba2293f4b45204825aed8efc7c4d9f1751f6891c94d192/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/175b1cade69506eba41a50a855002a44ad0fde9aec6b168db07158e19c72b578/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/82511632f85ca7ea82087ae4fb3c65a8036254cb70b33322cf6571595eea5cf6/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/177ee3c3e34a3807c4a55d8e824e6ffbc4ee8bfc5fa352aff16d8e22a9609fd9/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/c726f87c5c8a707656646425732483f10e317c365bde5c19ae6194e3cdda98e6/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/e555ab5e9152ac0b7e3c2187a920eb9523d485717453a06f452f9d8fbefe248b/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/418a028aed90f75dba47e9e9cef9539fc667e18045a95ff5853e5a6139b6ae73/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/0342390b772505e66643901e298ca3e50486e607c197344c1acc269eedf3ef88/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/73b3583b96118a8fc9472343cf411277f4adc79fcce3cd5e3053a0e4fc69470a/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/150170065c9c487134b6ff86d78b27ba473289de16038a3882bb7d15e2a1809f/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/455852614f74bedbe94b3914db2c24358f3d33e78711461da3468fed27157570/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/412a342aac067086612fe966e70b0fc76effaa12563eb0e082e90444874a00fe/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/e6cbbb5f2013720fef0f2a902f58513ed4d294f145e8e17046fcd2c98bdac526/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0fc7b9326d0e80220e28ed09adf6d5fc88e02f36bb3e95fa1e0bb3c1dc601afa/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f360cb0fda2d7170eb82fa789a67dc29df013656751067226198ce478611c3d9/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f90c0bef9992440e99d1efdaa60196d8ece162c0ffede8c9e9bb0b9e1a360c00/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/45407d2529a31a48b7b8d6a8d424f41da43fdc17551dbb136dccf34e8f85c975/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/259ee37e22ec695e459dc3782ca9fe1babcebc8309f85c9565ed79de97faeaed/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/772d55b9f07559a5e59ae754c27391454e38c550906afd16487608d05aa7d411/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/2e055160a51928b59868e355a08e569b6a9d96bcb0e093a959bb3552c71674a4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/565fc19b042c2d0764886a700ca39b3e898b67a1e62e72c7690952ce7b4c774a/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a6e1443f7adb4c81298ec64ecee3460129a36400a529e99845cd5f77ed1be071/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e26844e261ab155f10a87062d3cd5641835881f7cd9371365d73636c7c1a95f1/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.228654    5250 manager.go:217] Machine: {Timestamp:2026-10-30 05:31:25.228087024 +0000 UTC m=+0.059124232 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108983 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a2740e28994a7b33f2b2d05d33f3 SystemUUID:ec24a274-0e28-994a-7b33-f2b2d05d33f3 BootID:9f0c087d-d324-4152-9a0d-702ab9f066ec Filesystems:[{Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adb28f6d-9926-4b6a-b37a-149ae4785e1a/volumes/kubernetes.io~projected/kube-api-access-kf8kh DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe52fdd1efdf4085b572c56f940fd14235d8cb1be3a3e84459df6b6c164ad45f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e3b6130-4545-4c00-ab54-0c40c7a906bb/volumes/kubernetes.io~projected/kube-api-access-b7bb4 DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~projected/kube-api-access-27l79 DeviceMajor:0 DeviceMinor:270 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:274 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:276 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/840883aa-27bd-49c8-a676-1d8336ec90c2/volumes/kubernetes.io~projected/kube-api-access-c6j95 DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~projected/kube-api-access-lvm9w DeviceMajor:0 DeviceMinor:278 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1710ec37388ba46b5e004e7cea4f0b74eb72c88ae2b795deae975ac891d63a7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c44b9e34b90be5c951572b3193574cf979d3b7c87cf6b201c13305f8372fc09/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/087e82f9-024f-4bcd-97cf-8b7f928b719a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf928504a563cea3e752b24a491b5a8ca88db0c13e126d1fe38998d0c6c1e6d5/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62ed862a2a771f7c861fd0b52d87b5fcb9805e8259b4a1dd61b34fcd9d17255a/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d0ef442-4160-4ab1-80f1-03b37851a061/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a605957857e77d7c9ca1148e326e72d51619efc12a18c6ed2a134841094ab14/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bfe50f1f15d3c1b8a2f6a059ce4c86eac2b127060fb2a768164f8fc155b9bd0/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64b35bb9-7f5b-4009-a6cf-15acaf83b02f/volumes/kubernetes.io~projected/kube-api-access-wx2c4 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79eeb1e61cb18987fe4ab482fbbb44165beb370a49b6a4130bf40bdd544a8ce4/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volumes/kubernetes.io~projected/kube-api-access-f8chr DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51684f59-d0c8-4b0e-8f60-977be1864e8c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e9dbf9926bdb9e9741a66e8cbe7e0e2fb0438702fc2a99ec9db630a08183943/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e6bf8f2095624f2453cd1741d8801b5838db0d6f7ae237041956acbb352ce0d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdd2baec5fd002bdb88c1a9db7ecd659bbd78a9a8accd057bd6eab1cd4203eb0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/180d488f-b7df-4f3b-a6fb-b3c69f264d5c/volumes/kubernetes.io~projected/kube-api-access-vlh8n DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdbc4e4-f579-4d7b-abc3-295b81e4f468/volumes/kubernetes.io~projected/kube-api-access-2nmnv DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81290ea5-5539-4ab9-9747-673e98c728d4/volumes/kubernetes.io~projected/kube-api-access-lwq48 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0b6012dc0cee2e0173d1c27ac78d3970fb85701d1e9cf7074195dd710a26d9c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5070dc7e-02b2-450d-a4f8-0f616fbabc99/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:275 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1e3b6130-4545-4c00-ab54-0c40c7a906bb/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b7c849b7b57f9a3ed10cfba9d508c1dd1982e5d378a7a467f057f695db60d52/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f7b59b23ad26a44fd3b90f1fad8f1c7748d69162bd0ecae3ee6b33c082be49c/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d421e45-b806-4fcb-b501-62bf9f4079c0/volumes/kubernetes.io~projected/kube-api-access-cdbbp DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~projected/kube-api-access-nkskp DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b90d7ebe-df62-49e8-97e7-103ba14cd129/volumes/kubernetes.io~projected/kube-api-access-4q7vq DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab696c88-f337-4fd8-a9d7-be141c4c421c/volumes/kubernetes.io~projected/kube-api-access-xw7x2 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/664945e3-2c17-4cdf-a60d-4c274b0c9006/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55ddd6fa2f4f279b34fc1061da2f63bf24e4530c320964a81c7a91d18636d1ce/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~projected/kube-api-access-nrkt6 DeviceMajor:0 DeviceMinor:277 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a581d4f8-dc97-47d9-a409-bd391c36e67f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:332 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fae5a73bc0bec1c8fe0868046ec1d65af718f76d314d329413413eff7ff5d17/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ff467ebb2cfce1d0fa7a2aa50f84e935dd4f08a8f877b7dbb746050c6e1382a/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86e0d6284c01e3ba8246caab4a79288509c1e8f3e698088c2891223c178048ce/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d0ef442-4160-4ab1-80f1-03b37851a061/volumes/kubernetes.io~projected/kube-api-access-b2r6d DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1adbf04ed7d16dcac431fceeb3edd585eef53a64a359136b99b0973d38b1d0e4/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d421e45-b806-4fcb-b501-62bf9f4079c0/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29c13fe87efff85650760063975938ca72c64f7615f22cc0afa736a35d15da46/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/180d488f-b7df-4f3b-a6fb-b3c69f264d5c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/883d9833-be84-411b-998e-cea7379b6263/volumes/kubernetes.io~projected/kube-api-access-b7c8g DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1299b210-215f-44c3-9d45-2fda350dc319/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:273 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ff467ebb2cfce1 MacAddress:3a:c4:c8:4f:70:ab Speed:10000 Mtu:8901} {Name:1adbf04ed7d16dc MacAddress:ae:18:08:ad:37:fc Speed:10000 Mtu:8901} {Name:1e9dbf9926bdb9e MacAddress:92:af:8e:31:30:b8 Speed:10000 Mtu:8901} {Name:29c13fe87efff85 MacAddress:aa:c3:e3:d4:e6:cb Speed:10000 Mtu:8901} {Name:4c44b9e34b90be5 MacAddress:e2:53:1c:bb:d7:c2 Speed:10000 Mtu:8901} {Name:55ddd6fa2f4f279 MacAddress:ce:4c:41:a8:12:ab Speed:10000 Mtu:8901} {Name:5a605957857e77d MacAddress:fe:bb:8e:e3:68:b0 Speed:10000 Mtu:8901} {Name:79eeb1e61cb1898 MacAddress:6a:5f:c9:31:b7:8e Speed:10000 Mtu:8901} {Name:7f7b59b23ad26a4 MacAddress:c6:7f:cf:0a:27:22 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ef:af:29:47 Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:ac:bd:15:5f:0b Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:ef:af:29:47 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c2:b8:06:65:f0:27 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:62:db:7a:39:ad:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.228985    5250 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229026    5250 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229309    5250 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229330    5250 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229454    5250 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229460    5250 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229471    5250 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229483    5250 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229556    5250 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229713    5250 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229810    5250 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229843    5250 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229852    5250 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229861    5250 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229870    5250 kubelet.go:405] "Adding apiserver pod source"
Oct 30 05:31:25.230364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.229879    5250 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 05:31:25.230999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.230651    5250 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 05:31:25.230999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.230663    5250 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 05:31:25.231152 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.231133    5250 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 05:31:25.233385 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233371    5250 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 05:31:25.233385 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233382    5250 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 05:31:25.233500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233480    5250 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 05:31:25.233661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233650    5250 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 05:31:25.233703 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233679    5250 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 05:31:25.233772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233763    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 05:31:25.233811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233777    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 05:31:25.233811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233783    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 05:31:25.233811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233789    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 05:31:25.233811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233796    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 05:31:25.233811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233803    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 05:31:25.233811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233808    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 05:31:25.233811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233813    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 05:31:25.234118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233820    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 05:31:25.234118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233826    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 05:31:25.234118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233839    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 05:31:25.234118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233847    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 05:31:25.234118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233872    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 05:31:25.234118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.233878    5250 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 05:31:25.237093 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.237079    5250 server.go:1263] "Started kubelet"
Oct 30 05:31:25.237260 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.237209    5250 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 05:31:25.237317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.237284    5250 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 05:31:25.237547 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.237514    5250 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 05:31:25.237632 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.237581    5250 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 05:31:25.237653 ip-10-0-130-194 systemd[1]: Started Kubernetes Kubelet.
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.240345    5250 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.242502    5250 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.242524    5250 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.242589    5250 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:15 +0000 UTC" deadline="2028-08-08 16:07:48.074905774 +0000 UTC"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.242607    5250 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.242619    5250 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:25.242639    5250 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.242732    5250 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.242608    5250 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15562h36m22.832300664s"
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.243592    5250 factory.go:55] Registering systemd factory
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.243609    5250 factory.go:223] Registration of the systemd container factory successfully
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.244519    5250 factory.go:153] Registering CRI-O factory
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.244532    5250 factory.go:223] Registration of the crio container factory successfully
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.244590    5250 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.244609    5250 factory.go:103] Registering Raw factory
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.244624    5250 manager.go:1199] Started watching for new ooms in manager
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.244931    5250 manager.go:319] Starting recovery of all containers
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.244983    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="180d488f-b7df-4f3b-a6fb-b3c69f264d5c" volumeName="kubernetes.io/configmap/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245019    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e3b6130-4545-4c00-ab54-0c40c7a906bb" volumeName="kubernetes.io/configmap/1e3b6130-4545-4c00-ab54-0c40c7a906bb-config-volume" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245032    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e3b6130-4545-4c00-ab54-0c40c7a906bb" volumeName="kubernetes.io/empty-dir/1e3b6130-4545-4c00-ab54-0c40c7a906bb-tmp-dir" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245044    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" volumeName="kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-binary-copy" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245055    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae" volumeName="kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-tuned" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245069    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="180d488f-b7df-4f3b-a6fb-b3c69f264d5c" volumeName="kubernetes.io/secret/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245082    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdbc4e4-f579-4d7b-abc3-295b81e4f468" volumeName="kubernetes.io/projected/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-kube-api-access-2nmnv" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245095    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5070dc7e-02b2-450d-a4f8-0f616fbabc99" volumeName="kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-metrics-client-ca" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245112    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5070dc7e-02b2-450d-a4f8-0f616fbabc99" volumeName="kubernetes.io/empty-dir/5070dc7e-02b2-450d-a4f8-0f616fbabc99-volume-directive-shadow" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245125    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab696c88-f337-4fd8-a9d7-be141c4c421c" volumeName="kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-daemon-config" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245137    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="180d488f-b7df-4f3b-a6fb-b3c69f264d5c" volumeName="kubernetes.io/empty-dir/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-data-volume" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245150    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b90d7ebe-df62-49e8-97e7-103ba14cd129" volumeName="kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245203    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1299b210-215f-44c3-9d45-2fda350dc319" volumeName="kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245215    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5070dc7e-02b2-450d-a4f8-0f616fbabc99" volumeName="kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245229    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="087e82f9-024f-4bcd-97cf-8b7f928b719a" volumeName="kubernetes.io/configmap/087e82f9-024f-4bcd-97cf-8b7f928b719a-konnectivity-ca" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245242    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" volumeName="kubernetes.io/projected/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-kube-api-access-wx2c4" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245256    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="180d488f-b7df-4f3b-a6fb-b3c69f264d5c" volumeName="kubernetes.io/projected/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-api-access-vlh8n" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245271    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e3b6130-4545-4c00-ab54-0c40c7a906bb" volumeName="kubernetes.io/projected/1e3b6130-4545-4c00-ab54-0c40c7a906bb-kube-api-access-b7bb4" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245282    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5070dc7e-02b2-450d-a4f8-0f616fbabc99" volumeName="kubernetes.io/projected/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-api-access-lvm9w" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245294    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" volumeName="kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 05:31:25.245281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245307    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="840883aa-27bd-49c8-a676-1d8336ec90c2" volumeName="kubernetes.io/configmap/840883aa-27bd-49c8-a676-1d8336ec90c2-serviceca" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245320    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5070dc7e-02b2-450d-a4f8-0f616fbabc99" volumeName="kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245334    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664945e3-2c17-4cdf-a60d-4c274b0c9006" volumeName="kubernetes.io/configmap/664945e3-2c17-4cdf-a60d-4c274b0c9006-metrics-client-ca" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245347    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664945e3-2c17-4cdf-a60d-4c274b0c9006" volumeName="kubernetes.io/projected/664945e3-2c17-4cdf-a60d-4c274b0c9006-kube-api-access-27l79" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245359    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d421e45-b806-4fcb-b501-62bf9f4079c0" volumeName="kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245374    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5070dc7e-02b2-450d-a4f8-0f616fbabc99" volumeName="kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245387    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51684f59-d0c8-4b0e-8f60-977be1864e8c" volumeName="kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-config" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245400    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae" volumeName="kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-tmp" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245416    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1299b210-215f-44c3-9d45-2fda350dc319" volumeName="kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-metrics-client-ca" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245428    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51684f59-d0c8-4b0e-8f60-977be1864e8c" volumeName="kubernetes.io/secret/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245442    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="840883aa-27bd-49c8-a676-1d8336ec90c2" volumeName="kubernetes.io/projected/840883aa-27bd-49c8-a676-1d8336ec90c2-kube-api-access-c6j95" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245456    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d421e45-b806-4fcb-b501-62bf9f4079c0" volumeName="kubernetes.io/projected/3d421e45-b806-4fcb-b501-62bf9f4079c0-kube-api-access-cdbbp" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245468    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="883d9833-be84-411b-998e-cea7379b6263" volumeName="kubernetes.io/configmap/883d9833-be84-411b-998e-cea7379b6263-metrics-client-ca" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245479    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="883d9833-be84-411b-998e-cea7379b6263" volumeName="kubernetes.io/projected/883d9833-be84-411b-998e-cea7379b6263-kube-api-access-b7c8g" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245491    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="883d9833-be84-411b-998e-cea7379b6263" volumeName="kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245506    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="883d9833-be84-411b-998e-cea7379b6263" volumeName="kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245520    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab696c88-f337-4fd8-a9d7-be141c4c421c" volumeName="kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-cni-binary-copy" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245533    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae" volumeName="kubernetes.io/projected/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-kube-api-access-nkskp" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245544    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1299b210-215f-44c3-9d45-2fda350dc319" volumeName="kubernetes.io/empty-dir/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-textfile" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245558    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664945e3-2c17-4cdf-a60d-4c274b0c9006" volumeName="kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245572    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="087e82f9-024f-4bcd-97cf-8b7f928b719a" volumeName="kubernetes.io/secret/087e82f9-024f-4bcd-97cf-8b7f928b719a-agent-certs" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245587    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1299b210-215f-44c3-9d45-2fda350dc319" volumeName="kubernetes.io/projected/1299b210-215f-44c3-9d45-2fda350dc319-kube-api-access-nrkt6" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245600    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d0ef442-4160-4ab1-80f1-03b37851a061" volumeName="kubernetes.io/secret/3d0ef442-4160-4ab1-80f1-03b37851a061-cert" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245613    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51684f59-d0c8-4b0e-8f60-977be1864e8c" volumeName="kubernetes.io/projected/51684f59-d0c8-4b0e-8f60-977be1864e8c-kube-api-access-f8chr" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245626    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664945e3-2c17-4cdf-a60d-4c274b0c9006" volumeName="kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245642    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81290ea5-5539-4ab9-9747-673e98c728d4" volumeName="kubernetes.io/empty-dir/81290ea5-5539-4ab9-9747-673e98c728d4-tmp-dir" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245654    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a581d4f8-dc97-47d9-a409-bd391c36e67f" volumeName="kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245666    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1299b210-215f-44c3-9d45-2fda350dc319" volumeName="kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-tls" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245677    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdbc4e4-f579-4d7b-abc3-295b81e4f468" volumeName="kubernetes.io/configmap/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-iptables-alerter-script" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245690    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51684f59-d0c8-4b0e-8f60-977be1864e8c" volumeName="kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245704    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51684f59-d0c8-4b0e-8f60-977be1864e8c" volumeName="kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-env-overrides" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245716    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab696c88-f337-4fd8-a9d7-be141c4c421c" volumeName="kubernetes.io/projected/ab696c88-f337-4fd8-a9d7-be141c4c421c-kube-api-access-xw7x2" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245727    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1299b210-215f-44c3-9d45-2fda350dc319" volumeName="kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245739    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e3b6130-4545-4c00-ab54-0c40c7a906bb" volumeName="kubernetes.io/secret/1e3b6130-4545-4c00-ab54-0c40c7a906bb-metrics-tls" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245754    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d0ef442-4160-4ab1-80f1-03b37851a061" volumeName="kubernetes.io/projected/3d0ef442-4160-4ab1-80f1-03b37851a061-kube-api-access-b2r6d" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245766    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" volumeName="kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245806    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81290ea5-5539-4ab9-9747-673e98c728d4" volumeName="kubernetes.io/projected/81290ea5-5539-4ab9-9747-673e98c728d4-kube-api-access-lwq48" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245822    5250 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb28f6d-9926-4b6a-b37a-149ae4785e1a" volumeName="kubernetes.io/projected/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kube-api-access-kf8kh" seLinuxMountContext=""
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245837    5250 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 05:31:25.247239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.245845    5250 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 05:31:25.265529 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.265498    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:31:25.266710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.266687    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:31:25.271053 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.271024    5250 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 05:31:25.272582 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.272555    5250 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 05:31:25.272706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.272580    5250 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 05:31:25.272765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.272711    5250 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 05:31:25.273006 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:25.272949    5250 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 05:31:25.287194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.287144    5250 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="458f517e6142713aa4c2b17e27629a6e7b3a340306a310b63f3d818988fd7d30" exitCode=0
Oct 30 05:31:25.287194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.287193    5250 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="e879ff480a7794b454c3c71fc6c6530310dd168b8eb5998c5ce49fe3a629c330" exitCode=0
Oct 30 05:31:25.287365 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.287200    5250 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="39b7e7c070db680b9794a5131bbb5865f7e65bf64e8a8af8abd846b395dec056" exitCode=0
Oct 30 05:31:25.287365 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.287206    5250 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="b8dda2135827c434fb6cf79e3b51fe0e01b5076438b67f5f1888720fc0829799" exitCode=0
Oct 30 05:31:25.287365 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.287211    5250 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="33e9fddc0f7dbd2cb056bd132f909f747fccf9baca9c047cd9bbef8dfa264516" exitCode=0
Oct 30 05:31:25.287365 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.287216    5250 generic.go:356] "Generic (PLEG): container finished" podID="64b35bb9-7f5b-4009-a6cf-15acaf83b02f" containerID="a89038f6a50d996404d3fe2052092ad057143ee8e7e01a5560c7074bd7857cd3" exitCode=0
Oct 30 05:31:25.291370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.291342    5250 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:31:25.291472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.291404    5250 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 05:31:25.292137 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.292120    5250 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:31:25.295332 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.295313    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:31:25.295683 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.295655    5250 generic.go:356] "Generic (PLEG): container finished" podID="51684f59-d0c8-4b0e-8f60-977be1864e8c" containerID="346fe73080056c3708496a92956c1fa391bb48b6b1dbb4dda2c62a77583f9844" exitCode=1
Oct 30 05:31:25.307118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.307093    5250 generic.go:356] "Generic (PLEG): container finished" podID="10023875eee95e78098f943c5150d3b0" containerID="1e2c546e3032523d4fb6ef754b23fe43a37e3a8765dd126e556744eec339d917" exitCode=0
Oct 30 05:31:25.309986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.309960    5250 generic.go:356] "Generic (PLEG): container finished" podID="1299b210-215f-44c3-9d45-2fda350dc319" containerID="d0edc61ff6205706fa8a6456876e2c3cd20ac5f8afbdcad274c74211923515db" exitCode=0
Oct 30 05:31:25.314685 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.314662    5250 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:31:25.316565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.316515    5250 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 05:31:25.338258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.338240    5250 manager.go:324] Recovery completed
Oct 30 05:31:25.356688 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356636    5250 cpu_manager.go:226] "Starting" policy="none"
Oct 30 05:31:25.356688 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356648    5250 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 05:31:25.356688 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356664    5250 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 05:31:25.356827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356771    5250 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 05:31:25.356827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356781    5250 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 05:31:25.356827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356799    5250 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 05:31:25.356827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356804    5250 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 05:31:25.356827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356809    5250 policy_none.go:50] "Start"
Oct 30 05:31:25.356827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356817    5250 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 05:31:25.356827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356825    5250 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 05:31:25.357007 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356943    5250 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 05:31:25.357007 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356950    5250 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 05:31:25.357007 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.356955    5250 policy_none.go:44] "Start"
Oct 30 05:31:25.361159 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.361132    5250 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 05:31:25.364203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.364172    5250 manager.go:341] "Starting Device Plugin manager"
Oct 30 05:31:25.364690 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:25.364218    5250 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 05:31:25.364734 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.364694    5250 server.go:87] "Starting device plugin registration server"
Oct 30 05:31:25.364947 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.364937    5250 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 05:31:25.364977 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.364948    5250 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 05:31:25.365051 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.365034    5250 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 05:31:25.365159 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.365127    5250 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 05:31:25.365159 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.365158    5250 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 05:31:25.365328 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.365164    5250 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 05:31:25.365328 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.365221    5250 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:31:25.36514926Z","UUID":"2c42f9a5-c366-428d-b823-22f0593316cb","Handler":null,"Name":"","Endpoint":""}
Oct 30 05:31:25.367270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.367250    5250 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 05:31:25.367600 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.367294    5250 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 05:31:25.369203 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:25.369159    5250 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 05:31:25.373156 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.373119    5250 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"]
Oct 30 05:31:25.373811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.373797    5250 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.373977 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.373939    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" event={"ID":"2fd355e41b5bc4e012c326b7e91eb1cb","Type":"ContainerStarted","Data":"6a5023bfa0dd0b082a72d539ff0347852ae9e6ef3353a3f87ff64be82d501b0c"}
Oct 30 05:31:25.374038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.373990    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" event={"ID":"2fd355e41b5bc4e012c326b7e91eb1cb","Type":"ContainerStarted","Data":"f1710ec37388ba46b5e004e7cea4f0b74eb72c88ae2b795deae975ac891d63a7"}
Oct 30 05:31:25.374038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.374010    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"10023875eee95e78098f943c5150d3b0","Type":"ContainerStarted","Data":"202599dc127fbe3808a3175498c48a312fa2f7d510fc14ebc32e1a22139dd484"}
Oct 30 05:31:25.374038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.374030    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"10023875eee95e78098f943c5150d3b0","Type":"ContainerDied","Data":"1e2c546e3032523d4fb6ef754b23fe43a37e3a8765dd126e556744eec339d917"}
Oct 30 05:31:25.374201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.374049    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"10023875eee95e78098f943c5150d3b0","Type":"ContainerStarted","Data":"0fae5a73bc0bec1c8fe0868046ec1d65af718f76d314d329413413eff7ff5d17"}
Oct 30 05:31:25.374266 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.374222    5250 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.398471 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.398451    5250 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 05:31:25.398555 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:25.398488    5250 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.398934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.398922    5250 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 05:31:25.398965 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:25.398954    5250 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.465595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.465570    5250 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 05:31:25.467494 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.467470    5250 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 05:31:25.467602 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.467505    5250 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 05:31:25.467602 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.467519    5250 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID"
Oct 30 05:31:25.467602 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.467567    5250 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.491558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.491535    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.491558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.491559    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fd355e41b5bc4e012c326b7e91eb1cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"2fd355e41b5bc4e012c326b7e91eb1cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.491709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.491576    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.507591 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.507564    5250 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.507687 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.507633    5250 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.592036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.592002    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.592036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.592024    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.592229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.592082    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.592229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.592102    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fd355e41b5bc4e012c326b7e91eb1cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"2fd355e41b5bc4e012c326b7e91eb1cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.592229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.592142    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fd355e41b5bc4e012c326b7e91eb1cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"2fd355e41b5bc4e012c326b7e91eb1cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:25.592229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:25.592171    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10023875eee95e78098f943c5150d3b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"10023875eee95e78098f943c5150d3b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:26.230534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.230501    5250 apiserver.go:52] "Watching apiserver"
Oct 30 05:31:26.287489 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.287457    5250 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 05:31:26.287987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.287964    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dklsf","openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6","openshift-ovn-kubernetes/ovnkube-node-rm44x","kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal","openshift-dns/dns-default-rvnfl","openshift-monitoring/node-exporter-9h24j","openshift-network-operator/iptables-alerter-9wsmq","kube-system/konnectivity-agent-6nsmc","openshift-network-diagnostics/network-check-target-njsfm","openshift-image-registry/node-ca-v25l9","kube-system/global-pull-secret-syncer-r6plj","openshift-multus/multus-additional-cni-plugins-sc5dg","openshift-multus/network-metrics-daemon-qtv7g","openshift-cluster-node-tuning-operator/tuned-9k9f6","openshift-ingress-canary/ingress-canary-9665q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7","openshift-dns/node-resolver-nlkm2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal","openshift-multus/multus-rk729","openshift-insights/insights-runtime-extractor-gszgm","openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"]
Oct 30 05:31:26.303500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.303475    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wnglw\""
Oct 30 05:31:26.303765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.303749    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 05:31:26.303890 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.303870    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 05:31:26.303968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.303882    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 05:31:26.303968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.303959    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.304328 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.304309    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 05:31:26.304328 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.304323    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.321281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.321254    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 05:31:26.321337 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.321327    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.321377 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.321336    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 05:31:26.321502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.321490    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 05:31:26.321548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.321522    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nnkr\""
Oct 30 05:31:26.321548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.321526    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-twkbr\""
Oct 30 05:31:26.322330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.322318    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.322398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.322388    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.323495 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.323474    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.323495 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.323486    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.323608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.323585    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 05:31:26.324080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.324055    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 05:31:26.324836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.324814    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 05:31:26.324945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.324930    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 05:31:26.325080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325064    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 05:31:26.325164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325151    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 05:31:26.325257 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325244    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 05:31:26.325314 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325303    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnf9b\""
Oct 30 05:31:26.325382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325371    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 05:31:26.325479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325467    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.325611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325598    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.325702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325690    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 05:31:26.325774 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.325762    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c4j6h\""
Oct 30 05:31:26.326335 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.326318    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c6sjw\""
Oct 30 05:31:26.326406 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.326398    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-88sr6\""
Oct 30 05:31:26.326472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.326466    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 05:31:26.326563 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.326550    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 05:31:26.326672 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.326659    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.326739 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.326727    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gcmgk\""
Oct 30 05:31:26.326927 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.326912    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.326973 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.326963    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.327031 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.327018    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.327117 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.327100    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.327213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.327168    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 05:31:26.327213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.327199    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.327321 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.327273    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 05:31:26.330167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.330143    5250 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:26.330296 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.330282    5250 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:26.333227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333212    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5cj6p\""
Oct 30 05:31:26.333432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333412    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 05:31:26.333475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333443    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt4k5\""
Oct 30 05:31:26.333475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333462    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.333537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333494    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.333719 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333700    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w87jr\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333723    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-897vl\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333726    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w4h88\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333741    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kz82h\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333754    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333757    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333771    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333788    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-76x48\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333815    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9q7xl\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333848    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333857    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 05:31:26.333878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333858    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.334254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.333972    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 05:31:26.334254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.334024    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 05:31:26.334254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.334032    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 05:31:26.334254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.334078    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 05:31:26.334254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.334081    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 05:31:26.335352 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.335334    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 05:31:26.335419 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.335339    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fl95n\""
Oct 30 05:31:26.335419 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.335361    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2znb\""
Oct 30 05:31:26.335578 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.335560    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 05:31:26.335644 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.335602    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 05:31:26.335820 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.335800    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wzzmx\""
Oct 30 05:31:26.335938 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.335926    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 05:31:26.344382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.344367    5250 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 05:31:26.352318 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.352305    5250 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 05:31:26.352389 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:26.352352    5250 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:26.352769 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.352753    5250 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 05:31:26.352808 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:26.352795    5250 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal"
Oct 30 05:31:26.395926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.395893    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-systemd-units\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.396058 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.395935    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:31:26.396058 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.395965    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7x2\" (UniqueName: \"kubernetes.io/projected/ab696c88-f337-4fd8-a9d7-be141c4c421c-kube-api-access-xw7x2\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.396058 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396013    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81290ea5-5539-4ab9-9747-673e98c728d4-hosts-file\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:31:26.396206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396137    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-kubelet\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.396206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396193    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-bin\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.396292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396224    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-netns\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.396292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396255    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.396292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396281    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.396292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396285    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d421e45-b806-4fcb-b501-62bf9f4079c0-metrics-certs\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:31:26.396455 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396435    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-lib-modules\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.396488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396467    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-script-lib\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.396529 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396485    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2c4\" (UniqueName: \"kubernetes.io/projected/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-kube-api-access-wx2c4\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.396529 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396485    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.396595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396545    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:31:26.396595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396579    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-device-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.396669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396596    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-kubernetes\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.396669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396611    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-socket-dir-parent\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.396669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396627    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-bin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.396849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396673    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2r6d\" (UniqueName: \"kubernetes.io/projected/3d0ef442-4160-4ab1-80f1-03b37851a061-kube-api-access-b2r6d\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:26.396849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396709    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.396849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396743    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-script-lib\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.396849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396828    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-run\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.397008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396858    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-kubelet-config\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:26.397008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396877    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.397008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396892    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.397008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396908    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-sys\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.397008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396939    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-log-socket\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.397008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396975    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664945e3-2c17-4cdf-a60d-4c274b0c9006-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:26.397008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.396992    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/087e82f9-024f-4bcd-97cf-8b7f928b719a-konnectivity-ca\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397027    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397048    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-tls\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397093    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-iptables-alerter-script\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397119    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-k8s-cni-cncf-io\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397147    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397171    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/087e82f9-024f-4bcd-97cf-8b7f928b719a-konnectivity-ca\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397219    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397251    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8kh\" (UniqueName: \"kubernetes.io/projected/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kube-api-access-kf8kh\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.397281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397270    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664945e3-2c17-4cdf-a60d-4c274b0c9006-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397298    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397306    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-iptables-alerter-script\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397359    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkt6\" (UniqueName: \"kubernetes.io/projected/1299b210-215f-44c3-9d45-2fda350dc319-kube-api-access-nrkt6\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397415    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397462    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-netns\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397489    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397522    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmnv\" (UniqueName: \"kubernetes.io/projected/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-kube-api-access-2nmnv\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397555    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-tls\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.397596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397572    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-hostroot\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397608    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d0ef442-4160-4ab1-80f1-03b37851a061-cert\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397638    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-wtmp\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397755    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-daemon-config\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397806    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840883aa-27bd-49c8-a676-1d8336ec90c2-host\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397833    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d0ef442-4160-4ab1-80f1-03b37851a061-cert\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397910    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/883d9833-be84-411b-998e-cea7379b6263-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397948    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-conf\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397983    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-daemon-config\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.397987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.397990    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-var-lib-kubelet\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398016    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398043    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-systemd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398068    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-node-log\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398099    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398127    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-tmp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398131    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/883d9833-be84-411b-998e-cea7379b6263-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398165    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-host-slash\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398248    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398256    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-multus\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398274    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-tmp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398288    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-multus-certs\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398330    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/087e82f9-024f-4bcd-97cf-8b7f928b719a-agent-certs\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:31:26.398497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398421    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398492    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398529    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/087e82f9-024f-4bcd-97cf-8b7f928b719a-agent-certs\") pod \"konnectivity-agent-6nsmc\" (UID: \"087e82f9-024f-4bcd-97cf-8b7f928b719a\") " pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398575    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398603    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-var-lib-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398629    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-ovn\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398654    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-etc-kubernetes\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398685    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cnibin\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398690    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398711    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq48\" (UniqueName: \"kubernetes.io/projected/81290ea5-5539-4ab9-9747-673e98c728d4-kube-api-access-lwq48\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398744    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398885    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/883d9833-be84-411b-998e-cea7379b6263-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398918    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-conf-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.398990    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysconfig\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399051    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-slash\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399089    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7bb4\" (UniqueName: \"kubernetes.io/projected/1e3b6130-4545-4c00-ab54-0c40c7a906bb-kube-api-access-b7bb4\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399146    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399233    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkskp\" (UniqueName: \"kubernetes.io/projected/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-kube-api-access-nkskp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399238    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.399311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399316    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a581d4f8-dc97-47d9-a409-bd391c36e67f-original-pull-secret\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399383    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-metrics-client-ca\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399427    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-etc-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399427    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664945e3-2c17-4cdf-a60d-4c274b0c9006-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399443    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-system-cni-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399459    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-sys\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399497    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-system-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399535    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-cnibin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399567    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b6130-4545-4c00-ab54-0c40c7a906bb-tmp-dir\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399574    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-metrics-client-ca\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399601    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvm9w\" (UniqueName: \"kubernetes.io/projected/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-api-access-lvm9w\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399661    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b6130-4545-4c00-ab54-0c40c7a906bb-tmp-dir\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399699    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-data-volume\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399726    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399808    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399842    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399888    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-data-volume\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399919    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-kubelet\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.399950    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6j95\" (UniqueName: \"kubernetes.io/projected/840883aa-27bd-49c8-a676-1d8336ec90c2-kube-api-access-c6j95\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400029    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.400060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400034    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400096    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-systemd\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400123    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-root\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400159    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-config\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400223    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8chr\" (UniqueName: \"kubernetes.io/projected/51684f59-d0c8-4b0e-8f60-977be1864e8c-kube-api-access-f8chr\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400274    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400308    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400332    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400362    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-cni-binary-copy\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400386    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5070dc7e-02b2-450d-a4f8-0f616fbabc99-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400413    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-textfile\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400439    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400453    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovnkube-config\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400461    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-host\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400502    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbbp\" (UniqueName: \"kubernetes.io/projected/3d421e45-b806-4fcb-b501-62bf9f4079c0-kube-api-access-cdbbp\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400519    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400530    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400542    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5070dc7e-02b2-450d-a4f8-0f616fbabc99-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400558    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-binary-copy\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400577    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab696c88-f337-4fd8-a9d7-be141c4c421c-cni-binary-copy\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400605    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-textfile\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400582    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400725    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-env-overrides\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400770    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cni-binary-copy\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400753    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7c8g\" (UniqueName: \"kubernetes.io/projected/883d9833-be84-411b-998e-cea7379b6263-kube-api-access-b7c8g\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400806    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-accelerators-collector-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400824    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovn-node-metrics-cert\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400840    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-crio-socket\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400856    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-tuned\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400876    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e3b6130-4545-4c00-ab54-0c40c7a906bb-metrics-tls\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400899    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/840883aa-27bd-49c8-a676-1d8336ec90c2-serviceca\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400924    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51684f59-d0c8-4b0e-8f60-977be1864e8c-env-overrides\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.400953    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-modprobe-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401070    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-accelerators-collector-config\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401082    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-netd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401072    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51684f59-d0c8-4b0e-8f60-977be1864e8c-ovn-node-metrics-cert\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401106    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-tuned\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401127    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b6130-4545-4c00-ab54-0c40c7a906bb-config-volume\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401141    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e3b6130-4545-4c00-ab54-0c40c7a906bb-metrics-tls\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:26.401154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401171    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81290ea5-5539-4ab9-9747-673e98c728d4-tmp-dir\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:31:26.402439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401304    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlh8n\" (UniqueName: \"kubernetes.io/projected/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-api-access-vlh8n\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.402439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401313    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81290ea5-5539-4ab9-9747-673e98c728d4-tmp-dir\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:31:26.402439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401315    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b6130-4545-4c00-ab54-0c40c7a906bb-config-volume\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:26.402439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401333    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27l79\" (UniqueName: \"kubernetes.io/projected/664945e3-2c17-4cdf-a60d-4c274b0c9006-kube-api-access-27l79\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:26.402439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401362    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-dbus\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:26.402439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401389    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.402439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401567    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/840883aa-27bd-49c8-a676-1d8336ec90c2-serviceca\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:31:26.402439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.401695    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.422651 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.422626    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2c4\" (UniqueName: \"kubernetes.io/projected/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-kube-api-access-wx2c4\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.424816 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.424796    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmnv\" (UniqueName: \"kubernetes.io/projected/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-kube-api-access-2nmnv\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:31:26.425191 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.425157    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq48\" (UniqueName: \"kubernetes.io/projected/81290ea5-5539-4ab9-9747-673e98c728d4-kube-api-access-lwq48\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:31:26.425468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.425453    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkt6\" (UniqueName: \"kubernetes.io/projected/1299b210-215f-44c3-9d45-2fda350dc319-kube-api-access-nrkt6\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.425790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.425770    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2r6d\" (UniqueName: \"kubernetes.io/projected/3d0ef442-4160-4ab1-80f1-03b37851a061-kube-api-access-b2r6d\") pod \"ingress-canary-9665q\" (UID: \"3d0ef442-4160-4ab1-80f1-03b37851a061\") " pod="openshift-ingress-canary/ingress-canary-9665q"
Oct 30 05:31:26.427056 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.427037    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7x2\" (UniqueName: \"kubernetes.io/projected/ab696c88-f337-4fd8-a9d7-be141c4c421c-kube-api-access-xw7x2\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.428978 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.427447    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7vq\" (UniqueName: \"kubernetes.io/projected/b90d7ebe-df62-49e8-97e7-103ba14cd129-kube-api-access-4q7vq\") pod \"network-check-target-njsfm\" (UID: \"b90d7ebe-df62-49e8-97e7-103ba14cd129\") " pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:31:26.429486 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.429469    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8kh\" (UniqueName: \"kubernetes.io/projected/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kube-api-access-kf8kh\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.455900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.455875    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7bb4\" (UniqueName: \"kubernetes.io/projected/1e3b6130-4545-4c00-ab54-0c40c7a906bb-kube-api-access-b7bb4\") pod \"dns-default-rvnfl\" (UID: \"1e3b6130-4545-4c00-ab54-0c40c7a906bb\") " pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:26.458940 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.458919    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6j95\" (UniqueName: \"kubernetes.io/projected/840883aa-27bd-49c8-a676-1d8336ec90c2-kube-api-access-c6j95\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:31:26.459390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.459369    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbbp\" (UniqueName: \"kubernetes.io/projected/3d421e45-b806-4fcb-b501-62bf9f4079c0-kube-api-access-cdbbp\") pod \"network-metrics-daemon-qtv7g\" (UID: \"3d421e45-b806-4fcb-b501-62bf9f4079c0\") " pod="openshift-multus/network-metrics-daemon-qtv7g"
Oct 30 05:31:26.459452 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.459437    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkskp\" (UniqueName: \"kubernetes.io/projected/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-kube-api-access-nkskp\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.461369 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.461344    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlh8n\" (UniqueName: \"kubernetes.io/projected/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-kube-api-access-vlh8n\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.461369 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.461360    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7c8g\" (UniqueName: \"kubernetes.io/projected/883d9833-be84-411b-998e-cea7379b6263-kube-api-access-b7c8g\") pod \"prometheus-operator-9cd89b74c-dklsf\" (UID: \"883d9833-be84-411b-998e-cea7379b6263\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dklsf"
Oct 30 05:31:26.461480 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.461384    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvm9w\" (UniqueName: \"kubernetes.io/projected/5070dc7e-02b2-450d-a4f8-0f616fbabc99-kube-api-access-lvm9w\") pod \"kube-state-metrics-5dc8d7d89b-mdrmp\" (UID: \"5070dc7e-02b2-450d-a4f8-0f616fbabc99\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mdrmp"
Oct 30 05:31:26.464093 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.464069    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8chr\" (UniqueName: \"kubernetes.io/projected/51684f59-d0c8-4b0e-8f60-977be1864e8c-kube-api-access-f8chr\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.464287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.464268    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27l79\" (UniqueName: \"kubernetes.io/projected/664945e3-2c17-4cdf-a60d-4c274b0c9006-kube-api-access-27l79\") pod \"openshift-state-metrics-7bd7489957-xdvj6\" (UID: \"664945e3-2c17-4cdf-a60d-4c274b0c9006\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xdvj6"
Oct 30 05:31:26.501926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.501865    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.501926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.501889    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-kubelet\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.501926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.501905    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-systemd\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.501926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.501920    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-root\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.501936    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.501952    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.501989    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-kubelet\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502011    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-systemd\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502020    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502013    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-root\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502035    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-host\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502023    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502065    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502067    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502092    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-host\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502111    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502119    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502130    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-crio-socket\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502147    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502156    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-modprobe-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502194    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-netd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502205    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180d488f-b7df-4f3b-a6fb-b3c69f264d5c-crio-socket\") pod \"insights-runtime-extractor-gszgm\" (UID: \"180d488f-b7df-4f3b-a6fb-b3c69f264d5c\") " pod="openshift-insights/insights-runtime-extractor-gszgm"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502216    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-dbus\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502237    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-netd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502239    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-systemd-units\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502244    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-modprobe-d\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502260    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81290ea5-5539-4ab9-9747-673e98c728d4-hosts-file\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502272    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-dbus\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502281    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-systemd-units\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502274    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-kubelet\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502295    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-kubelet\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502323    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81290ea5-5539-4ab9-9747-673e98c728d4-hosts-file\") pod \"node-resolver-nlkm2\" (UID: \"81290ea5-5539-4ab9-9747-673e98c728d4\") " pod="openshift-dns/node-resolver-nlkm2"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502326    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-bin\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502346    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-netns\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502362    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502369    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-cni-bin\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502375    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-lib-modules\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502408    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-device-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502433    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-kubernetes\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502457    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-socket-dir-parent\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502460    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-device-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502460    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-netns\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502462    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-lib-modules\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502479    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502480    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-bin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502494    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-kubernetes\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502504    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-socket-dir-parent\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502530    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502564    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-run\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502580    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-bin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502592    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502607    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-kubelet-config\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502643    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-run\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502649    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a581d4f8-dc97-47d9-a409-bd391c36e67f-kubelet-config\") pod \"global-pull-secret-syncer-r6plj\" (UID: \"a581d4f8-dc97-47d9-a409-bd391c36e67f\") " pod="kube-system/global-pull-secret-syncer-r6plj"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502674    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502697    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502703    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.502714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502730    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-sys\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502749    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-log-socket\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502777    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-log-socket\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502782    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-sys\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502787    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-k8s-cni-cncf-io\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502807    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-k8s-cni-cncf-io\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502820    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adb28f6d-9926-4b6a-b37a-149ae4785e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-tzxf7\" (UID: \"adb28f6d-9926-4b6a-b37a-149ae4785e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzxf7"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502825    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-netns\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502853    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-hostroot\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502856    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-netns\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502877    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-wtmp\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502892    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840883aa-27bd-49c8-a676-1d8336ec90c2-host\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502896    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-hostroot\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502906    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-conf\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502920    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-var-lib-kubelet\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502935    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-systemd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502940    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-node-exporter-wtmp\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502956    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-node-log\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502965    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840883aa-27bd-49c8-a676-1d8336ec90c2-host\") pod \"node-ca-v25l9\" (UID: \"840883aa-27bd-49c8-a676-1d8336ec90c2\") " pod="openshift-image-registry/node-ca-v25l9"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502981    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502991    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-node-log\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502970    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-systemd\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.502982    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-var-lib-kubelet\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503008    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-host-slash\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503014    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503030    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysctl-conf\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503034    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-multus\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503052    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdbc4e4-f579-4d7b-abc3-295b81e4f468-host-slash\") pod \"iptables-alerter-9wsmq\" (UID: \"4bdbc4e4-f579-4d7b-abc3-295b81e4f468\") " pod="openshift-network-operator/iptables-alerter-9wsmq"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503057    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-multus-certs\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503079    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-var-lib-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503090    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-var-lib-cni-multus\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503093    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-ovn\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503113    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-host-run-multus-certs\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503120    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-etc-kubernetes\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503115    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-run-ovn\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503135    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cnibin\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503153    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-conf-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503156    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-etc-kubernetes\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503194    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-multus-conf-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503134    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-var-lib-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503194    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysconfig\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503243    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-slash\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503259    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-etc-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503226    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae-etc-sysconfig\") pod \"tuned-9k9f6\" (UID: \"bd29a4f1-41e1-4fb2-aef3-2b0abf8f8bae\") " pod="openshift-cluster-node-tuning-operator/tuned-9k9f6"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503212    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-cnibin\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503282    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-system-cni-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503298    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-etc-openvswitch\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503309    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-sys\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503318    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51684f59-d0c8-4b0e-8f60-977be1864e8c-host-slash\") pod \"ovnkube-node-rm44x\" (UID: \"51684f59-d0c8-4b0e-8f60-977be1864e8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503326    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-system-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503339    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-cnibin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503357    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1299b210-215f-44c3-9d45-2fda350dc319-sys\") pod \"node-exporter-9h24j\" (UID: \"1299b210-215f-44c3-9d45-2fda350dc319\") " pod="openshift-monitoring/node-exporter-9h24j"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503340    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-system-cni-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503362    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503407    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-cnibin\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503409    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64b35bb9-7f5b-4009-a6cf-15acaf83b02f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc5dg\" (UID: \"64b35bb9-7f5b-4009-a6cf-15acaf83b02f\") " pod="openshift-multus/multus-additional-cni-plugins-sc5dg"
Oct 30 05:31:26.503806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:26.503408    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab696c88-f337-4fd8-a9d7-be141c4c421c-system-cni-dir\") pod \"multus-rk729\" (UID: \"ab696c88-f337-4fd8-a9d7-be141c4c421c\") " pod="openshift-multus/multus-rk729"
Oct 30 05:31:28.375951 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:28.375918    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rvnfl"
Oct 30 05:31:29.423056 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:29.423023    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:31:29.526849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:29.526800    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:29.711074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:29.710984    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:29.885663 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:29.885633    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:31.062786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:31.062756    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:31.692129 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:31.692103    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:31.692323 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:31.692311    5250 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 05:31:32.373019 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:31:32.372984    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:31:37.202585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:37.202498    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:37.371240 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:37.371208    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:44.118535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:44.118505    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm44x"
Oct 30 05:31:46.244988 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:46.244946    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:46.407766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:46.407731    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:47.570207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:47.570164    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6nsmc"
Oct 30 05:31:52.167695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:52.167664    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:52.317678 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:52.317642    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:55.103195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:55.103148    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:55.250331 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:55.250295    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:55.426828 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:55.426796    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:55.567067 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:55.567027    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:59.425861 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:31:59.425830    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njsfm"
Oct 30 05:32:02.371572 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:32:02.371536    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:32:29.764158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:29.764122    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:32.371629 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:32:32.371589    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:32:42.169078 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:42.169043    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:42.309730 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:42.309683    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:42.843327 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:42.843291    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:42.989382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:42.989351    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:44.512424 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:44.512380    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:44.661647 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:44.661609    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:54.921659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:54.921626    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:55.834641 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:32:55.834604    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:02.371343 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:33:02.371306    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:33:07.263415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:33:07.263338    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:07.416799 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:33:07.416759    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:29.900297 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:33:29.900261    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:32.371834 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:33:32.371798    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:34:02.371599 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:34:02.371562    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:34:18.078672 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:34:18.078638    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:25.182854 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:34:25.182824    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:30.012427 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:34:30.012381    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:32.371691 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:34:32.371651    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:35:02.371798 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:35:02.371720    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:35:28.597496 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:28.597460    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:28.766040 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:28.766011    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:29.366679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.366650    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"]
Oct 30 05:35:29.369941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.369923    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.399408 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.399387    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 05:35:29.399408 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.399402    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 05:35:29.399630 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.399452    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-x6bg6\""
Oct 30 05:35:29.400403 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.400383    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"]
Oct 30 05:35:29.431127 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.431098    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.431264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.431133    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.431264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.431154    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7whs\" (UniqueName: \"kubernetes.io/projected/d37bd12c-b76d-4a7c-b714-a2ee006c3015-kube-api-access-k7whs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.531590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.531564    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.531736 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.531598    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.531736 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.531625    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7whs\" (UniqueName: \"kubernetes.io/projected/d37bd12c-b76d-4a7c-b714-a2ee006c3015-kube-api-access-k7whs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.531929 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.531896    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.532005 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.531955    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.554592 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.554567    5250 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 05:35:29.556873 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.556848    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7whs\" (UniqueName: \"kubernetes.io/projected/d37bd12c-b76d-4a7c-b714-a2ee006c3015-kube-api-access-k7whs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.685288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.685261    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:29.983331 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.983250    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"]
Oct 30 05:35:29.987824 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:35:29.987791    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37bd12c_b76d_4a7c_b714_a2ee006c3015.slice/crio-c3c92b0b490966d91e8f433fb0dd8f3c6314ab3360f219a46bbc91554baaeec8 WatchSource:0}: Error finding container c3c92b0b490966d91e8f433fb0dd8f3c6314ab3360f219a46bbc91554baaeec8: Status 404 returned error can't find the container with id c3c92b0b490966d91e8f433fb0dd8f3c6314ab3360f219a46bbc91554baaeec8
Oct 30 05:35:29.989967 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:29.989947    5250 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:35:30.073020 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:30.072989    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k" event={"ID":"d37bd12c-b76d-4a7c-b714-a2ee006c3015","Type":"ContainerStarted","Data":"c3c92b0b490966d91e8f433fb0dd8f3c6314ab3360f219a46bbc91554baaeec8"}
Oct 30 05:35:30.128999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:30.128961    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:32.377999 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:35:32.377949    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:35:36.097258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:36.097226    5250 generic.go:356] "Generic (PLEG): container finished" podID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerID="7798689a6713f0043622e73a0c64abb4e4000098e56bab351ec1f808b3078dd3" exitCode=0
Oct 30 05:35:36.097665 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:36.097280    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k" event={"ID":"d37bd12c-b76d-4a7c-b714-a2ee006c3015","Type":"ContainerDied","Data":"7798689a6713f0043622e73a0c64abb4e4000098e56bab351ec1f808b3078dd3"}
Oct 30 05:35:39.109268 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:39.109234    5250 generic.go:356] "Generic (PLEG): container finished" podID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerID="3dc0c2d49c76e04aa88f857aab0f4e7b682d9246b051982bcce76250651fac18" exitCode=0
Oct 30 05:35:39.109713 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:39.109274    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k" event={"ID":"d37bd12c-b76d-4a7c-b714-a2ee006c3015","Type":"ContainerDied","Data":"3dc0c2d49c76e04aa88f857aab0f4e7b682d9246b051982bcce76250651fac18"}
Oct 30 05:35:45.130683 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:45.130604    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k" event={"ID":"d37bd12c-b76d-4a7c-b714-a2ee006c3015","Type":"ContainerStarted","Data":"e2d5bf419f05d4dbbf4010e45feec7f9d3f07320d4d7940417329932f0bd9d7e"}
Oct 30 05:35:45.176173 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:45.176125    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k" podStartSLOduration=3.206167258 podStartE2EDuration="16.176110533s" totalImagesPullingTime="12.969943275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:35:45.175505076 +0000 UTC m=+260.006542292" watchObservedRunningTime="2026-10-30 05:35:45.176110533 +0000 UTC m=+260.007147772"
Oct 30 05:35:46.135908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:46.135871    5250 generic.go:356] "Generic (PLEG): container finished" podID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerID="e2d5bf419f05d4dbbf4010e45feec7f9d3f07320d4d7940417329932f0bd9d7e" exitCode=0
Oct 30 05:35:46.136316 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:46.135955    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k" event={"ID":"d37bd12c-b76d-4a7c-b714-a2ee006c3015","Type":"ContainerDied","Data":"e2d5bf419f05d4dbbf4010e45feec7f9d3f07320d4d7940417329932f0bd9d7e"}
Oct 30 05:35:47.261637 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.261610    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:47.344834 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.344808    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-bundle\") pod \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") "
Oct 30 05:35:47.345004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.344845    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-util\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-util\") pod \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") "
Oct 30 05:35:47.345004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.344878    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37bd12c-b76d-4a7c-b714-a2ee006c3015-kube-api-access-k7whs\" (UniqueName: \"kubernetes.io/projected/d37bd12c-b76d-4a7c-b714-a2ee006c3015-kube-api-access-k7whs\") pod \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\" (UID: \"d37bd12c-b76d-4a7c-b714-a2ee006c3015\") "
Oct 30 05:35:47.345628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.345549    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-bundle" pod "d37bd12c-b76d-4a7c-b714-a2ee006c3015" (UID: "d37bd12c-b76d-4a7c-b714-a2ee006c3015"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:47.347274 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.347248    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37bd12c-b76d-4a7c-b714-a2ee006c3015-kube-api-access-k7whs" pod "d37bd12c-b76d-4a7c-b714-a2ee006c3015" (UID: "d37bd12c-b76d-4a7c-b714-a2ee006c3015"). InnerVolumeSpecName "kube-api-access-k7whs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:47.348677 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.348656    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-util" pod "d37bd12c-b76d-4a7c-b714-a2ee006c3015" (UID: "d37bd12c-b76d-4a7c-b714-a2ee006c3015"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:47.446378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.446289    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7whs\" (UniqueName: \"kubernetes.io/projected/d37bd12c-b76d-4a7c-b714-a2ee006c3015-kube-api-access-k7whs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:35:47.446378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.446327    5250 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-bundle\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:35:47.446378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:47.446337    5250 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d37bd12c-b76d-4a7c-b714-a2ee006c3015-util\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:35:48.144392 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:48.144368    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k"
Oct 30 05:35:48.144559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:48.144362    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2672k" event={"ID":"d37bd12c-b76d-4a7c-b714-a2ee006c3015","Type":"ContainerDied","Data":"c3c92b0b490966d91e8f433fb0dd8f3c6314ab3360f219a46bbc91554baaeec8"}
Oct 30 05:35:48.144559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:48.144488    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c92b0b490966d91e8f433fb0dd8f3c6314ab3360f219a46bbc91554baaeec8"
Oct 30 05:35:50.671390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:50.671358    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:52.073207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.073157    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:52.634962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.634917    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"]
Oct 30 05:35:52.635708 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.635686    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerName="util"
Oct 30 05:35:52.635840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.635827    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerName="util"
Oct 30 05:35:52.635923 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.635913    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerName="extract"
Oct 30 05:35:52.635994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.635985    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerName="extract"
Oct 30 05:35:52.636064 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.636056    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerName="pull"
Oct 30 05:35:52.636128 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.636120    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerName="pull"
Oct 30 05:35:52.636370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.636356    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37bd12c-b76d-4a7c-b714-a2ee006c3015" containerName="extract"
Oct 30 05:35:52.665237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.665204    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"]
Oct 30 05:35:52.665378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.665312    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"
Oct 30 05:35:52.681511 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.681487    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/101b5e82-c3a0-4040-95c2-94b2f8045823-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-bcg2r\" (UID: \"101b5e82-c3a0-4040-95c2-94b2f8045823\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"
Oct 30 05:35:52.681608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.681527    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5622\" (UniqueName: \"kubernetes.io/projected/101b5e82-c3a0-4040-95c2-94b2f8045823-kube-api-access-h5622\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-bcg2r\" (UID: \"101b5e82-c3a0-4040-95c2-94b2f8045823\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"
Oct 30 05:35:52.681608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.681539    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 05:35:52.681843 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.681829    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 05:35:52.681993 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.681977    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7dr4l\""
Oct 30 05:35:52.782188 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.782157    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/101b5e82-c3a0-4040-95c2-94b2f8045823-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-bcg2r\" (UID: \"101b5e82-c3a0-4040-95c2-94b2f8045823\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"
Oct 30 05:35:52.782317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.782223    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5622\" (UniqueName: \"kubernetes.io/projected/101b5e82-c3a0-4040-95c2-94b2f8045823-kube-api-access-h5622\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-bcg2r\" (UID: \"101b5e82-c3a0-4040-95c2-94b2f8045823\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"
Oct 30 05:35:52.782533 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.782512    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/101b5e82-c3a0-4040-95c2-94b2f8045823-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-bcg2r\" (UID: \"101b5e82-c3a0-4040-95c2-94b2f8045823\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"
Oct 30 05:35:52.807377 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.807356    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5622\" (UniqueName: \"kubernetes.io/projected/101b5e82-c3a0-4040-95c2-94b2f8045823-kube-api-access-h5622\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-bcg2r\" (UID: \"101b5e82-c3a0-4040-95c2-94b2f8045823\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"
Oct 30 05:35:52.995237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:52.995129    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"
Oct 30 05:35:53.339488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:53.339407    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r"]
Oct 30 05:35:53.343133 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:35:53.343100    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101b5e82_c3a0_4040_95c2_94b2f8045823.slice/crio-339e345dcf77125a175d2d5cdc0d9270f533cb698be3a78095e4b88b68e521f3 WatchSource:0}: Error finding container 339e345dcf77125a175d2d5cdc0d9270f533cb698be3a78095e4b88b68e521f3: Status 404 returned error can't find the container with id 339e345dcf77125a175d2d5cdc0d9270f533cb698be3a78095e4b88b68e521f3
Oct 30 05:35:54.078061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:54.078010    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:54.167527 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:54.167495    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r" event={"ID":"101b5e82-c3a0-4040-95c2-94b2f8045823","Type":"ContainerStarted","Data":"339e345dcf77125a175d2d5cdc0d9270f533cb698be3a78095e4b88b68e521f3"}
Oct 30 05:35:54.226749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:54.226708    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:56.175100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:56.175069    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r" event={"ID":"101b5e82-c3a0-4040-95c2-94b2f8045823","Type":"ContainerStarted","Data":"7c2eac9dc639db02924703510781181624e5a0107346d154c5e40635bea56ecc"}
Oct 30 05:35:56.256429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:56.256374    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-bcg2r" podStartSLOduration=2.233809847 podStartE2EDuration="4.256355s" totalImagesPullingTime="2.022545153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:35:56.240664543 +0000 UTC m=+271.071701759" watchObservedRunningTime="2026-10-30 05:35:56.256355 +0000 UTC m=+271.087392211"
Oct 30 05:35:56.467095 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:56.467002    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:56.649235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:56.649195    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:56.863926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:56.863842    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:57.078794 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.078760    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:57.390959 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.390928    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"]
Oct 30 05:35:57.409430 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.409406    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:35:57.416924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.416897    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"]
Oct 30 05:35:57.429997 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.429974    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-bwx84\""
Oct 30 05:35:57.433405 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.433377    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 05:35:57.439440 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.439425    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 05:35:57.502396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.502376    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"]
Oct 30 05:35:57.517403 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.517380    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj8b8\" (UniqueName: \"kubernetes.io/projected/a55f3b2c-a554-448d-8455-d4bd9c51aff9-kube-api-access-pj8b8\") pod \"cert-manager-webhook-5dbbbf75cc-drcjl\" (UID: \"a55f3b2c-a554-448d-8455-d4bd9c51aff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:35:57.517491 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.517412    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a55f3b2c-a554-448d-8455-d4bd9c51aff9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-drcjl\" (UID: \"a55f3b2c-a554-448d-8455-d4bd9c51aff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:35:57.527062 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.527039    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"]
Oct 30 05:35:57.527148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.527129    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.543589 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.543572    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-x57qz\""
Oct 30 05:35:57.543723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.543700    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 05:35:57.543805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.543794    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 05:35:57.618058 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.618038    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.618198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.618072    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8b8\" (UniqueName: \"kubernetes.io/projected/a55f3b2c-a554-448d-8455-d4bd9c51aff9-kube-api-access-pj8b8\") pod \"cert-manager-webhook-5dbbbf75cc-drcjl\" (UID: \"a55f3b2c-a554-448d-8455-d4bd9c51aff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:35:57.618198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.618095    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a55f3b2c-a554-448d-8455-d4bd9c51aff9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-drcjl\" (UID: \"a55f3b2c-a554-448d-8455-d4bd9c51aff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:35:57.618198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.618117    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2wg\" (UniqueName: \"kubernetes.io/projected/64391419-aa2f-452e-9a03-0653747a9421-kube-api-access-zc2wg\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.618319 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.618255    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.643860 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.643795    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a55f3b2c-a554-448d-8455-d4bd9c51aff9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-drcjl\" (UID: \"a55f3b2c-a554-448d-8455-d4bd9c51aff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:35:57.643967 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.643901    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8b8\" (UniqueName: \"kubernetes.io/projected/a55f3b2c-a554-448d-8455-d4bd9c51aff9-kube-api-access-pj8b8\") pod \"cert-manager-webhook-5dbbbf75cc-drcjl\" (UID: \"a55f3b2c-a554-448d-8455-d4bd9c51aff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:35:57.719156 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.719128    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2wg\" (UniqueName: \"kubernetes.io/projected/64391419-aa2f-452e-9a03-0653747a9421-kube-api-access-zc2wg\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.719280 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.719171    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.719280 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.719236    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.719493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.719469    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.719599 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.719536    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.741293 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.741268    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:35:57.746370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.746351    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2wg\" (UniqueName: \"kubernetes.io/projected/64391419-aa2f-452e-9a03-0653747a9421-kube-api-access-zc2wg\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:57.879582 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:57.879548    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:35:58.059286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:58.059254    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"]
Oct 30 05:35:58.062033 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:35:58.061992    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55f3b2c_a554_448d_8455_d4bd9c51aff9.slice/crio-baa99407c6338999cfcbcfa283290afcb689dc9f601523c75856f90eba34c929 WatchSource:0}: Error finding container baa99407c6338999cfcbcfa283290afcb689dc9f601523c75856f90eba34c929: Status 404 returned error can't find the container with id baa99407c6338999cfcbcfa283290afcb689dc9f601523c75856f90eba34c929
Oct 30 05:35:58.183670 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:58.183604    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl" event={"ID":"a55f3b2c-a554-448d-8455-d4bd9c51aff9","Type":"ContainerStarted","Data":"baa99407c6338999cfcbcfa283290afcb689dc9f601523c75856f90eba34c929"}
Oct 30 05:35:58.196004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:58.195981    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"]
Oct 30 05:35:58.199358 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:35:58.199329    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64391419_aa2f_452e_9a03_0653747a9421.slice/crio-6d1fa91dd3b7549d9b9b326b5918710012b8a3b60286bee0edfb320b1c325cb2 WatchSource:0}: Error finding container 6d1fa91dd3b7549d9b9b326b5918710012b8a3b60286bee0edfb320b1c325cb2: Status 404 returned error can't find the container with id 6d1fa91dd3b7549d9b9b326b5918710012b8a3b60286bee0edfb320b1c325cb2
Oct 30 05:35:58.857350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:58.857317    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:59.091333 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:59.091294    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:59.190669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:59.190640    5250 generic.go:356] "Generic (PLEG): container finished" podID="64391419-aa2f-452e-9a03-0653747a9421" containerID="b4cf14414c95baef86f6e82f8537d2bce30fe327397cd719408c7e3eb0f3b7c3" exitCode=0
Oct 30 05:35:59.190860 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:59.190699    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4" event={"ID":"64391419-aa2f-452e-9a03-0653747a9421","Type":"ContainerDied","Data":"b4cf14414c95baef86f6e82f8537d2bce30fe327397cd719408c7e3eb0f3b7c3"}
Oct 30 05:35:59.190860 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:35:59.190785    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4" event={"ID":"64391419-aa2f-452e-9a03-0653747a9421","Type":"ContainerStarted","Data":"6d1fa91dd3b7549d9b9b326b5918710012b8a3b60286bee0edfb320b1c325cb2"}
Oct 30 05:36:00.878041 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:00.878002    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:01.026110 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:01.026031    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:01.201542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:01.201506    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl" event={"ID":"a55f3b2c-a554-448d-8455-d4bd9c51aff9","Type":"ContainerStarted","Data":"37c97d95c2cc95caba42c79cf65e3aef65bc3b0ebe45825d4cb572a998a9f766"}
Oct 30 05:36:01.201722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:01.201643    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:36:01.245145 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:01.245090    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl" podStartSLOduration=1.380019506 podStartE2EDuration="4.245073399s" totalImagesPullingTime="2.865053893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:01.244528226 +0000 UTC m=+276.075565443" watchObservedRunningTime="2026-10-30 05:36:01.245073399 +0000 UTC m=+276.076110616"
Oct 30 05:36:02.207371 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:02.207336    5250 generic.go:356] "Generic (PLEG): container finished" podID="64391419-aa2f-452e-9a03-0653747a9421" containerID="e16d0187d92c35a0165f213f58a806c7adaf0e27164e33b444ac0b2c010ccf0f" exitCode=0
Oct 30 05:36:02.207852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:02.207419    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4" event={"ID":"64391419-aa2f-452e-9a03-0653747a9421","Type":"ContainerDied","Data":"e16d0187d92c35a0165f213f58a806c7adaf0e27164e33b444ac0b2c010ccf0f"}
Oct 30 05:36:02.372488 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:36:02.372452    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:36:03.213922 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:03.213887    5250 generic.go:356] "Generic (PLEG): container finished" podID="64391419-aa2f-452e-9a03-0653747a9421" containerID="37d017fbddba7481706adb339de4a23c980072e56c543aa5d45d45abd0f675d2" exitCode=0
Oct 30 05:36:03.214328 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:03.213964    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4" event={"ID":"64391419-aa2f-452e-9a03-0653747a9421","Type":"ContainerDied","Data":"37d017fbddba7481706adb339de4a23c980072e56c543aa5d45d45abd0f675d2"}
Oct 30 05:36:03.968727 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:03.968695    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"]
Oct 30 05:36:03.971535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:03.971520    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"
Oct 30 05:36:04.004312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.004282    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xzw2v\""
Oct 30 05:36:04.004810 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.004791    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"]
Oct 30 05:36:04.066470 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.066447    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a9a12cc-d00d-4089-aa4f-717a4ccfcb15-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8dzrv\" (UID: \"7a9a12cc-d00d-4089-aa4f-717a4ccfcb15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"
Oct 30 05:36:04.066603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.066485    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zwq\" (UniqueName: \"kubernetes.io/projected/7a9a12cc-d00d-4089-aa4f-717a4ccfcb15-kube-api-access-z5zwq\") pod \"cert-manager-cainjector-5c57bbd4fc-8dzrv\" (UID: \"7a9a12cc-d00d-4089-aa4f-717a4ccfcb15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"
Oct 30 05:36:04.167243 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.167215    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a9a12cc-d00d-4089-aa4f-717a4ccfcb15-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8dzrv\" (UID: \"7a9a12cc-d00d-4089-aa4f-717a4ccfcb15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"
Oct 30 05:36:04.167424 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.167256    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zwq\" (UniqueName: \"kubernetes.io/projected/7a9a12cc-d00d-4089-aa4f-717a4ccfcb15-kube-api-access-z5zwq\") pod \"cert-manager-cainjector-5c57bbd4fc-8dzrv\" (UID: \"7a9a12cc-d00d-4089-aa4f-717a4ccfcb15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"
Oct 30 05:36:04.191930 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.191907    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a9a12cc-d00d-4089-aa4f-717a4ccfcb15-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8dzrv\" (UID: \"7a9a12cc-d00d-4089-aa4f-717a4ccfcb15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"
Oct 30 05:36:04.193808 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.193784    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zwq\" (UniqueName: \"kubernetes.io/projected/7a9a12cc-d00d-4089-aa4f-717a4ccfcb15-kube-api-access-z5zwq\") pod \"cert-manager-cainjector-5c57bbd4fc-8dzrv\" (UID: \"7a9a12cc-d00d-4089-aa4f-717a4ccfcb15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"
Oct 30 05:36:04.282665 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.282639    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"
Oct 30 05:36:04.347791 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.347446    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:36:04.369568 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.369544    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-bundle\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-bundle\") pod \"64391419-aa2f-452e-9a03-0653747a9421\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") "
Oct 30 05:36:04.369688 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.369623    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64391419-aa2f-452e-9a03-0653747a9421-kube-api-access-zc2wg\" (UniqueName: \"kubernetes.io/projected/64391419-aa2f-452e-9a03-0653747a9421-kube-api-access-zc2wg\") pod \"64391419-aa2f-452e-9a03-0653747a9421\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") "
Oct 30 05:36:04.369688 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.369662    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-util\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-util\") pod \"64391419-aa2f-452e-9a03-0653747a9421\" (UID: \"64391419-aa2f-452e-9a03-0653747a9421\") "
Oct 30 05:36:04.369975 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.369942    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-bundle" pod "64391419-aa2f-452e-9a03-0653747a9421" (UID: "64391419-aa2f-452e-9a03-0653747a9421"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:04.371699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.371679    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64391419-aa2f-452e-9a03-0653747a9421-kube-api-access-zc2wg" pod "64391419-aa2f-452e-9a03-0653747a9421" (UID: "64391419-aa2f-452e-9a03-0653747a9421"). InnerVolumeSpecName "kube-api-access-zc2wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:04.374399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.374362    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-util" pod "64391419-aa2f-452e-9a03-0653747a9421" (UID: "64391419-aa2f-452e-9a03-0653747a9421"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:04.470395 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.470367    5250 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-bundle\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:36:04.470395 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.470391    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2wg\" (UniqueName: \"kubernetes.io/projected/64391419-aa2f-452e-9a03-0653747a9421-kube-api-access-zc2wg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:36:04.470395 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.470401    5250 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64391419-aa2f-452e-9a03-0653747a9421-util\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:36:04.584310 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:04.584236    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv"]
Oct 30 05:36:04.587281 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:36:04.587252    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a9a12cc_d00d_4089_aa4f_717a4ccfcb15.slice/crio-cbcbe5c9b74e5f0240bfa82858c46c219bcca05fc369ed068ef46dcf8b249af0 WatchSource:0}: Error finding container cbcbe5c9b74e5f0240bfa82858c46c219bcca05fc369ed068ef46dcf8b249af0: Status 404 returned error can't find the container with id cbcbe5c9b74e5f0240bfa82858c46c219bcca05fc369ed068ef46dcf8b249af0
Oct 30 05:36:05.223306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:05.223277    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4"
Oct 30 05:36:05.223306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:05.223289    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc852hkz4" event={"ID":"64391419-aa2f-452e-9a03-0653747a9421","Type":"ContainerDied","Data":"6d1fa91dd3b7549d9b9b326b5918710012b8a3b60286bee0edfb320b1c325cb2"}
Oct 30 05:36:05.223507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:05.223372    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d1fa91dd3b7549d9b9b326b5918710012b8a3b60286bee0edfb320b1c325cb2"
Oct 30 05:36:05.224827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:05.224797    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv" event={"ID":"7a9a12cc-d00d-4089-aa4f-717a4ccfcb15","Type":"ContainerStarted","Data":"3875b356b1e1cbd6e7a6cba1ae7a3a26dcf0b4f93c80705bf70dc4b0cc058309"}
Oct 30 05:36:05.224932 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:05.224838    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv" event={"ID":"7a9a12cc-d00d-4089-aa4f-717a4ccfcb15","Type":"ContainerStarted","Data":"cbcbe5c9b74e5f0240bfa82858c46c219bcca05fc369ed068ef46dcf8b249af0"}
Oct 30 05:36:05.267451 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:05.267410    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8dzrv" podStartSLOduration=2.267395031 podStartE2EDuration="2.267395031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:05.266864717 +0000 UTC m=+280.097901934" watchObservedRunningTime="2026-10-30 05:36:05.267395031 +0000 UTC m=+280.098432249"
Oct 30 05:36:05.617530 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:05.617444    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:05.758830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:05.758754    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:06.105027 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.104995    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:06.912748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.912716    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-m6d8t"]
Oct 30 05:36:06.913169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.913001    5250 cpu_manager.go:414] "containerMap: removing container" podUID="64391419-aa2f-452e-9a03-0653747a9421" containerName="util"
Oct 30 05:36:06.913169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.913013    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64391419-aa2f-452e-9a03-0653747a9421" containerName="util"
Oct 30 05:36:06.913169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.913027    5250 cpu_manager.go:414] "containerMap: removing container" podUID="64391419-aa2f-452e-9a03-0653747a9421" containerName="pull"
Oct 30 05:36:06.913169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.913034    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64391419-aa2f-452e-9a03-0653747a9421" containerName="pull"
Oct 30 05:36:06.913169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.913041    5250 cpu_manager.go:414] "containerMap: removing container" podUID="64391419-aa2f-452e-9a03-0653747a9421" containerName="extract"
Oct 30 05:36:06.913169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.913046    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64391419-aa2f-452e-9a03-0653747a9421" containerName="extract"
Oct 30 05:36:06.913169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.913101    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="64391419-aa2f-452e-9a03-0653747a9421" containerName="extract"
Oct 30 05:36:06.915886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.915871    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-m6d8t"
Oct 30 05:36:06.947943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.947919    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-svfsm\""
Oct 30 05:36:06.948634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.948613    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-m6d8t"]
Oct 30 05:36:06.987558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.987522    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc929426-1d00-4612-af6e-40210ef3012c-bound-sa-token\") pod \"cert-manager-597b95cb84-m6d8t\" (UID: \"cc929426-1d00-4612-af6e-40210ef3012c\") " pod="cert-manager/cert-manager-597b95cb84-m6d8t"
Oct 30 05:36:06.987558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:06.987557    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ktw\" (UniqueName: \"kubernetes.io/projected/cc929426-1d00-4612-af6e-40210ef3012c-kube-api-access-h9ktw\") pod \"cert-manager-597b95cb84-m6d8t\" (UID: \"cc929426-1d00-4612-af6e-40210ef3012c\") " pod="cert-manager/cert-manager-597b95cb84-m6d8t"
Oct 30 05:36:07.088472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:07.088438    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc929426-1d00-4612-af6e-40210ef3012c-bound-sa-token\") pod \"cert-manager-597b95cb84-m6d8t\" (UID: \"cc929426-1d00-4612-af6e-40210ef3012c\") " pod="cert-manager/cert-manager-597b95cb84-m6d8t"
Oct 30 05:36:07.088472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:07.088472    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ktw\" (UniqueName: \"kubernetes.io/projected/cc929426-1d00-4612-af6e-40210ef3012c-kube-api-access-h9ktw\") pod \"cert-manager-597b95cb84-m6d8t\" (UID: \"cc929426-1d00-4612-af6e-40210ef3012c\") " pod="cert-manager/cert-manager-597b95cb84-m6d8t"
Oct 30 05:36:07.113193 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:07.113160    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc929426-1d00-4612-af6e-40210ef3012c-bound-sa-token\") pod \"cert-manager-597b95cb84-m6d8t\" (UID: \"cc929426-1d00-4612-af6e-40210ef3012c\") " pod="cert-manager/cert-manager-597b95cb84-m6d8t"
Oct 30 05:36:07.113347 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:07.113329    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ktw\" (UniqueName: \"kubernetes.io/projected/cc929426-1d00-4612-af6e-40210ef3012c-kube-api-access-h9ktw\") pod \"cert-manager-597b95cb84-m6d8t\" (UID: \"cc929426-1d00-4612-af6e-40210ef3012c\") " pod="cert-manager/cert-manager-597b95cb84-m6d8t"
Oct 30 05:36:07.209682 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:07.209608    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl"
Oct 30 05:36:07.226224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:07.226171    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-m6d8t"
Oct 30 05:36:07.532625 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:07.532552    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-m6d8t"]
Oct 30 05:36:07.535850 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:36:07.535822    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc929426_1d00_4612_af6e_40210ef3012c.slice/crio-9c4d592d7b4e4cbce9ae27b7e7ded94117d92b5695f7ba28816bdd5394ef0894 WatchSource:0}: Error finding container 9c4d592d7b4e4cbce9ae27b7e7ded94117d92b5695f7ba28816bdd5394ef0894: Status 404 returned error can't find the container with id 9c4d592d7b4e4cbce9ae27b7e7ded94117d92b5695f7ba28816bdd5394ef0894
Oct 30 05:36:08.236780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:08.236751    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-m6d8t" event={"ID":"cc929426-1d00-4612-af6e-40210ef3012c","Type":"ContainerStarted","Data":"2773887729e877bc556d5a8d62f7e35e72b169ee9310abee8f6b1b03674e83be"}
Oct 30 05:36:08.237207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:08.236789    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-m6d8t" event={"ID":"cc929426-1d00-4612-af6e-40210ef3012c","Type":"ContainerStarted","Data":"9c4d592d7b4e4cbce9ae27b7e7ded94117d92b5695f7ba28816bdd5394ef0894"}
Oct 30 05:36:13.709890 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:13.709849    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:13.858054 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:13.858013    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:20.739694 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:20.739657    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:20.879253 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:20.879223    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:22.321341 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.321304    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:22.844914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.844866    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-m6d8t" podStartSLOduration=16.84485331 podStartE2EDuration="16.84485331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:08.291128823 +0000 UTC m=+283.122166039" watchObservedRunningTime="2026-10-30 05:36:22.84485331 +0000 UTC m=+297.675890526"
Oct 30 05:36:22.845424 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.845399    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"]
Oct 30 05:36:22.848810 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.848788    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:22.867843 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.867821    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"]
Oct 30 05:36:22.880780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.880755    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-x9658\""
Oct 30 05:36:22.880901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.880762    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 05:36:22.880901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.880762    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 05:36:22.901866 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.901839    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kpwf\" (UniqueName: \"kubernetes.io/projected/4da3cb68-1fad-438c-ac60-7bc243d99b23-kube-api-access-6kpwf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:22.901944 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.901882    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:22.901944 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:22.901910    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:23.002564 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.002537    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpwf\" (UniqueName: \"kubernetes.io/projected/4da3cb68-1fad-438c-ac60-7bc243d99b23-kube-api-access-6kpwf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:23.002665 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.002588    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:23.002720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.002692    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:23.002935 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.002921    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:23.002999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.002985    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:23.027677 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.027651    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpwf\" (UniqueName: \"kubernetes.io/projected/4da3cb68-1fad-438c-ac60-7bc243d99b23-kube-api-access-6kpwf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:23.147934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.147900    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:23.164772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.164749    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:23.298849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.298812    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:23.462224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:23.462123    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"]
Oct 30 05:36:23.465244 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:36:23.465165    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da3cb68_1fad_438c_ac60_7bc243d99b23.slice/crio-21ca6f3c3e754ff955ebbfd58db19e1641a2e6ea6af75b8dfba0aca11507d9e0 WatchSource:0}: Error finding container 21ca6f3c3e754ff955ebbfd58db19e1641a2e6ea6af75b8dfba0aca11507d9e0: Status 404 returned error can't find the container with id 21ca6f3c3e754ff955ebbfd58db19e1641a2e6ea6af75b8dfba0aca11507d9e0
Oct 30 05:36:24.303656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:24.303573    5250 generic.go:356] "Generic (PLEG): container finished" podID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerID="956c462be96a60e19b799e1419fe761ffb2871c8c7fd04bc7345d8aee17e1ea0" exitCode=0
Oct 30 05:36:24.303656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:24.303616    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4" event={"ID":"4da3cb68-1fad-438c-ac60-7bc243d99b23","Type":"ContainerDied","Data":"956c462be96a60e19b799e1419fe761ffb2871c8c7fd04bc7345d8aee17e1ea0"}
Oct 30 05:36:24.303855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:24.303664    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4" event={"ID":"4da3cb68-1fad-438c-ac60-7bc243d99b23","Type":"ContainerStarted","Data":"21ca6f3c3e754ff955ebbfd58db19e1641a2e6ea6af75b8dfba0aca11507d9e0"}
Oct 30 05:36:25.296295 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:25.296272    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:36:25.296800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:25.296782    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:36:25.311256 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:25.311226    5250 generic.go:356] "Generic (PLEG): container finished" podID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerID="b771c6a01039a6a2959085c37f5ea0215a38358646e624c24c34535f2536b881" exitCode=0
Oct 30 05:36:25.311348 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:25.311267    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4" event={"ID":"4da3cb68-1fad-438c-ac60-7bc243d99b23","Type":"ContainerDied","Data":"b771c6a01039a6a2959085c37f5ea0215a38358646e624c24c34535f2536b881"}
Oct 30 05:36:26.318012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:26.317973    5250 generic.go:356] "Generic (PLEG): container finished" podID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerID="382ed8fcc83d355e0c7cdaf4ec614e6e2b9e02051da876ee389c647dd392e953" exitCode=0
Oct 30 05:36:26.318436 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:26.318055    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4" event={"ID":"4da3cb68-1fad-438c-ac60-7bc243d99b23","Type":"ContainerDied","Data":"382ed8fcc83d355e0c7cdaf4ec614e6e2b9e02051da876ee389c647dd392e953"}
Oct 30 05:36:27.448881 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.448860    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:27.535269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.535242    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-bundle\") pod \"4da3cb68-1fad-438c-ac60-7bc243d99b23\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") "
Oct 30 05:36:27.535407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.535333    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-util\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-util\") pod \"4da3cb68-1fad-438c-ac60-7bc243d99b23\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") "
Oct 30 05:36:27.535407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.535357    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da3cb68-1fad-438c-ac60-7bc243d99b23-kube-api-access-6kpwf\" (UniqueName: \"kubernetes.io/projected/4da3cb68-1fad-438c-ac60-7bc243d99b23-kube-api-access-6kpwf\") pod \"4da3cb68-1fad-438c-ac60-7bc243d99b23\" (UID: \"4da3cb68-1fad-438c-ac60-7bc243d99b23\") "
Oct 30 05:36:27.535811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.535776    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-bundle" pod "4da3cb68-1fad-438c-ac60-7bc243d99b23" (UID: "4da3cb68-1fad-438c-ac60-7bc243d99b23"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:27.537604 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.537576    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da3cb68-1fad-438c-ac60-7bc243d99b23-kube-api-access-6kpwf" pod "4da3cb68-1fad-438c-ac60-7bc243d99b23" (UID: "4da3cb68-1fad-438c-ac60-7bc243d99b23"). InnerVolumeSpecName "kube-api-access-6kpwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:27.541223 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.541201    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-util" pod "4da3cb68-1fad-438c-ac60-7bc243d99b23" (UID: "4da3cb68-1fad-438c-ac60-7bc243d99b23"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:27.635862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.635838    5250 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-util\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:36:27.635862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.635862    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kpwf\" (UniqueName: \"kubernetes.io/projected/4da3cb68-1fad-438c-ac60-7bc243d99b23-kube-api-access-6kpwf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:36:27.636032 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:27.635873    5250 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da3cb68-1fad-438c-ac60-7bc243d99b23-bundle\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:36:28.328528 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:28.328501    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4"
Oct 30 05:36:28.328528 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:28.328511    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpzz4" event={"ID":"4da3cb68-1fad-438c-ac60-7bc243d99b23","Type":"ContainerDied","Data":"21ca6f3c3e754ff955ebbfd58db19e1641a2e6ea6af75b8dfba0aca11507d9e0"}
Oct 30 05:36:28.328760 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:28.328547    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21ca6f3c3e754ff955ebbfd58db19e1641a2e6ea6af75b8dfba0aca11507d9e0"
Oct 30 05:36:30.273346 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:30.273313    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:32.371524 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:36:32.371484    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:36:32.816998 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.816968    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:32.943703 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.943670    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"]
Oct 30 05:36:32.943941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.943929    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerName="extract"
Oct 30 05:36:32.944025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.943942    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerName="extract"
Oct 30 05:36:32.944025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.943952    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerName="pull"
Oct 30 05:36:32.944025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.943957    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerName="pull"
Oct 30 05:36:32.944025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.943965    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerName="util"
Oct 30 05:36:32.944025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.943970    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerName="util"
Oct 30 05:36:32.944025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.944023    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da3cb68-1fad-438c-ac60-7bc243d99b23" containerName="extract"
Oct 30 05:36:32.948075 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.948056    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:32.977396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:32.977365    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"]
Oct 30 05:36:33.033161 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.033133    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:33.047758 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.047735    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 05:36:33.047949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.047809    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zvlk9\""
Oct 30 05:36:33.047949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.047911    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 05:36:33.069301 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.069241    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 05:36:33.069407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.069399    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 05:36:33.069496 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.069480    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 05:36:33.077354 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.077336    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/103e8ad9-d682-4fef-b195-11ca136dc5a4-cert\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.077424 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.077375    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/103e8ad9-d682-4fef-b195-11ca136dc5a4-metrics-cert\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.077466 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.077399    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hkr\" (UniqueName: \"kubernetes.io/projected/103e8ad9-d682-4fef-b195-11ca136dc5a4-kube-api-access-k6hkr\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.077519 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.077504    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/103e8ad9-d682-4fef-b195-11ca136dc5a4-manager-config\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.178802 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.178767    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/103e8ad9-d682-4fef-b195-11ca136dc5a4-metrics-cert\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.178802 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.178800    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hkr\" (UniqueName: \"kubernetes.io/projected/103e8ad9-d682-4fef-b195-11ca136dc5a4-kube-api-access-k6hkr\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.179044 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.178835    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/103e8ad9-d682-4fef-b195-11ca136dc5a4-manager-config\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.179044 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.178871    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/103e8ad9-d682-4fef-b195-11ca136dc5a4-cert\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.179596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.179572    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/103e8ad9-d682-4fef-b195-11ca136dc5a4-manager-config\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.181520 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.181501    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/103e8ad9-d682-4fef-b195-11ca136dc5a4-cert\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.181570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.181544    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/103e8ad9-d682-4fef-b195-11ca136dc5a4-metrics-cert\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.188540 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.188517    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:33.207395 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.207366    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hkr\" (UniqueName: \"kubernetes.io/projected/103e8ad9-d682-4fef-b195-11ca136dc5a4-kube-api-access-k6hkr\") pod \"lws-controller-manager-58867fb74d-t9zk4\" (UID: \"103e8ad9-d682-4fef-b195-11ca136dc5a4\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.259616 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.259573    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:33.413355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.413325    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:33.558163 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.558134    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"]
Oct 30 05:36:33.561009 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:36:33.560985    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103e8ad9_d682_4fef_b195_11ca136dc5a4.slice/crio-160ee55b5d0b3db3819fe433517b5d4a7a2fe68cf54ec2b1e2089c3b8681171c WatchSource:0}: Error finding container 160ee55b5d0b3db3819fe433517b5d4a7a2fe68cf54ec2b1e2089c3b8681171c: Status 404 returned error can't find the container with id 160ee55b5d0b3db3819fe433517b5d4a7a2fe68cf54ec2b1e2089c3b8681171c
Oct 30 05:36:33.567981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.567957    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:33.881672 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:33.881581    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:34.099747 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.099712    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"]
Oct 30 05:36:34.104285 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.104267    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.139118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.139094    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"]
Oct 30 05:36:34.171495 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.171473    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:34.189606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.189579    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.189723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.189613    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2cba753f-bbea-4075-9a7b-7648d2e13965-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.189723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.189633    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.189723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.189652    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.189848 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.189759    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlhv\" (UniqueName: \"kubernetes.io/projected/2cba753f-bbea-4075-9a7b-7648d2e13965-kube-api-access-5hlhv\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.189848 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.189819    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cba753f-bbea-4075-9a7b-7648d2e13965-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.210934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.210909    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 05:36:34.211050 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.211005    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 05:36:34.211251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.211222    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 05:36:34.211251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.211237    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wrhw7\""
Oct 30 05:36:34.211443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.211267    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 05:36:34.211443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.211325    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 05:36:34.290384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.290353    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2cba753f-bbea-4075-9a7b-7648d2e13965-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.290500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.290386    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.290500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.290410    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.290500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.290428    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlhv\" (UniqueName: \"kubernetes.io/projected/2cba753f-bbea-4075-9a7b-7648d2e13965-kube-api-access-5hlhv\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.290500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.290479    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cba753f-bbea-4075-9a7b-7648d2e13965-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.290706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.290523    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.290853 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.290832    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2cba753f-bbea-4075-9a7b-7648d2e13965-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.292914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.292891    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cba753f-bbea-4075-9a7b-7648d2e13965-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.293012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.292997    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.293209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.293173    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.293282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.293261    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2cba753f-bbea-4075-9a7b-7648d2e13965-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.315215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.315168    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlhv\" (UniqueName: \"kubernetes.io/projected/2cba753f-bbea-4075-9a7b-7648d2e13965-kube-api-access-5hlhv\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9\" (UID: \"2cba753f-bbea-4075-9a7b-7648d2e13965\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.326827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.326799    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:34.352201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.352154    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4" event={"ID":"103e8ad9-d682-4fef-b195-11ca136dc5a4","Type":"ContainerStarted","Data":"160ee55b5d0b3db3819fe433517b5d4a7a2fe68cf54ec2b1e2089c3b8681171c"}
Oct 30 05:36:34.419338 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.419270    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:34.476122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.475402    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:34.635319 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.635280    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:34.775518 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.775444    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"]
Oct 30 05:36:34.778594 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:36:34.778566    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cba753f_bbea_4075_9a7b_7648d2e13965.slice/crio-496a0efbb97744dd6b4ff781006d35376708cc70ea53f37f457dd6300aba4b5b WatchSource:0}: Error finding container 496a0efbb97744dd6b4ff781006d35376708cc70ea53f37f457dd6300aba4b5b: Status 404 returned error can't find the container with id 496a0efbb97744dd6b4ff781006d35376708cc70ea53f37f457dd6300aba4b5b
Oct 30 05:36:34.787780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:34.787755    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:35.358864 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:35.358825    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9" event={"ID":"2cba753f-bbea-4075-9a7b-7648d2e13965","Type":"ContainerStarted","Data":"496a0efbb97744dd6b4ff781006d35376708cc70ea53f37f457dd6300aba4b5b"}
Oct 30 05:36:36.365080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:36.365036    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4" event={"ID":"103e8ad9-d682-4fef-b195-11ca136dc5a4","Type":"ContainerStarted","Data":"dd3dcd7a4c46472997eb2caa9979ba53a82ca6c5f012b8a2e6f287639c045638"}
Oct 30 05:36:36.365551 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:36.365212    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:36.409866 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:36.409810    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4" podStartSLOduration=2.577311732 podStartE2EDuration="4.40979373s" totalImagesPullingTime="1.832481998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:36.409137673 +0000 UTC m=+311.240174904" watchObservedRunningTime="2026-10-30 05:36:36.40979373 +0000 UTC m=+311.240830950"
Oct 30 05:36:38.374499 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:38.374465    5250 generic.go:356] "Generic (PLEG): container finished" podID="2cba753f-bbea-4075-9a7b-7648d2e13965" containerID="3a08c48535da1a42ac276b6f648be621319adaa14be52930ee7183e408dfbb88" exitCode=0
Oct 30 05:36:38.374931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:38.374535    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9" event={"ID":"2cba753f-bbea-4075-9a7b-7648d2e13965","Type":"ContainerDied","Data":"3a08c48535da1a42ac276b6f648be621319adaa14be52930ee7183e408dfbb88"}
Oct 30 05:36:39.380844 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:39.380806    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9" event={"ID":"2cba753f-bbea-4075-9a7b-7648d2e13965","Type":"ContainerStarted","Data":"914456b47de6f76f5b668f9d6ed63d8472b18365655f8089921f423524a2d76a"}
Oct 30 05:36:39.381249 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:39.380897    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:39.531322 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:39.531274    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9" podStartSLOduration=1.917837611 podStartE2EDuration="5.531261008s" totalImagesPullingTime="3.180355095s" totalInitContainerRuntime="433.068302ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:39.51763057 +0000 UTC m=+314.348667788" watchObservedRunningTime="2026-10-30 05:36:39.531261008 +0000 UTC m=+314.362298288"
Oct 30 05:36:47.371688 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:47.371654    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-t9zk4"
Oct 30 05:36:50.387919 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:50.387881    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9"
Oct 30 05:36:53.379960 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:53.379919    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:53.535577 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:53.535546    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:55.883633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:55.883601    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:56.035942 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:56.035910    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:57.512309 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:57.512264    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:57.680354 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:57.680321    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:58.922200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:58.922146    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:59.070856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.070796    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:59.158301 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.158267    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-jbpqr"]
Oct 30 05:36:59.164577 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.164554    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:36:59.180298 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.180256    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-jbpqr"]
Oct 30 05:36:59.192604 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.192582    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 05:36:59.192730 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.192627    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-ncpml\""
Oct 30 05:36:59.267396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.267353    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:59.280234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.280204    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p855\" (UniqueName: \"kubernetes.io/projected/945cda97-a3c0-4488-a5d1-5bc921afb74f-kube-api-access-2p855\") pod \"odh-observability-584f99c788-jbpqr\" (UID: \"945cda97-a3c0-4488-a5d1-5bc921afb74f\") " pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:36:59.280407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.280262    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/945cda97-a3c0-4488-a5d1-5bc921afb74f-webhook-certs\") pod \"odh-observability-584f99c788-jbpqr\" (UID: \"945cda97-a3c0-4488-a5d1-5bc921afb74f\") " pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:36:59.380991 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.380957    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p855\" (UniqueName: \"kubernetes.io/projected/945cda97-a3c0-4488-a5d1-5bc921afb74f-kube-api-access-2p855\") pod \"odh-observability-584f99c788-jbpqr\" (UID: \"945cda97-a3c0-4488-a5d1-5bc921afb74f\") " pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:36:59.380991 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.380995    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/945cda97-a3c0-4488-a5d1-5bc921afb74f-webhook-certs\") pod \"odh-observability-584f99c788-jbpqr\" (UID: \"945cda97-a3c0-4488-a5d1-5bc921afb74f\") " pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:36:59.381220 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:36:59.381123    5250 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 05:36:59.381220 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:36:59.381215    5250 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945cda97-a3c0-4488-a5d1-5bc921afb74f-webhook-certs podName:945cda97-a3c0-4488-a5d1-5bc921afb74f nodeName:}" failed. No retries permitted until 2026-10-30 05:36:59.881199899 +0000 UTC m=+334.712237107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/945cda97-a3c0-4488-a5d1-5bc921afb74f-webhook-certs") pod "odh-observability-584f99c788-jbpqr" (UID: "945cda97-a3c0-4488-a5d1-5bc921afb74f") : secret "odh-observability-webhook-cert" not found
Oct 30 05:36:59.406210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.406159    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p855\" (UniqueName: \"kubernetes.io/projected/945cda97-a3c0-4488-a5d1-5bc921afb74f-kube-api-access-2p855\") pod \"odh-observability-584f99c788-jbpqr\" (UID: \"945cda97-a3c0-4488-a5d1-5bc921afb74f\") " pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:36:59.732362 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.732326    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:59.885198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.885155    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/945cda97-a3c0-4488-a5d1-5bc921afb74f-webhook-certs\") pod \"odh-observability-584f99c788-jbpqr\" (UID: \"945cda97-a3c0-4488-a5d1-5bc921afb74f\") " pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:36:59.887725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:36:59.887696    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/945cda97-a3c0-4488-a5d1-5bc921afb74f-webhook-certs\") pod \"odh-observability-584f99c788-jbpqr\" (UID: \"945cda97-a3c0-4488-a5d1-5bc921afb74f\") " pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:37:00.084320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:00.084230    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:37:00.400535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:00.400510    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-jbpqr"]
Oct 30 05:37:00.403061 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:37:00.403024    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945cda97_a3c0_4488_a5d1_5bc921afb74f.slice/crio-fd04a2ec5f9f8306208bc753e9280f466a589bbf7cadad31a4c14c527c097165 WatchSource:0}: Error finding container fd04a2ec5f9f8306208bc753e9280f466a589bbf7cadad31a4c14c527c097165: Status 404 returned error can't find the container with id fd04a2ec5f9f8306208bc753e9280f466a589bbf7cadad31a4c14c527c097165
Oct 30 05:37:00.464120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:00.464092    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-jbpqr" event={"ID":"945cda97-a3c0-4488-a5d1-5bc921afb74f","Type":"ContainerStarted","Data":"fd04a2ec5f9f8306208bc753e9280f466a589bbf7cadad31a4c14c527c097165"}
Oct 30 05:37:01.687259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.687216    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:01.848251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.848211    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:01.908212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.907093    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"]
Oct 30 05:37:01.910757 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.910735    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:01.932237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.932209    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tlchv\""
Oct 30 05:37:01.945831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.945724    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 05:37:01.945831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.945774    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 05:37:01.945831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.945771    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 05:37:01.946756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:01.946733    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"]
Oct 30 05:37:02.001634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.001601    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.001800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.001652    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.001800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.001682    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.001918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.001812    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.001918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.001854    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.001918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.001898    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.002076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.002002    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm2zf\" (UniqueName: \"kubernetes.io/projected/db89190c-4734-40d6-a4e3-ab0ca9c35cec-kube-api-access-xm2zf\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.002076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.002042    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.002076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.002063    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.038986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.038946    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:02.103014 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.102973    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103014 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103015    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103285 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103045    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103285 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103071    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103285 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103107    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103285 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103143    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm2zf\" (UniqueName: \"kubernetes.io/projected/db89190c-4734-40d6-a4e3-ab0ca9c35cec-kube-api-access-xm2zf\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103285 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103203    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103285 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103233    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103292    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103419    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103487    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103526    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103757 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103661    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.103918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.103899    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.106239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.106214    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.107483 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.107462    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.128494 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.128455    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db89190c-4734-40d6-a4e3-ab0ca9c35cec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.128815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.128793    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm2zf\" (UniqueName: \"kubernetes.io/projected/db89190c-4734-40d6-a4e3-ab0ca9c35cec-kube-api-access-xm2zf\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb54lwql\" (UID: \"db89190c-4734-40d6-a4e3-ab0ca9c35cec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.226775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.226699    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:02.248055 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.248012    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:02.379504 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:37:02.379472    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:37:02.397121 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.397084    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:02.558484 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:02.558412    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"]
Oct 30 05:37:02.561493 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:37:02.561465    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb89190c_4734_40d6_a4e3_ab0ca9c35cec.slice/crio-5eeaa6619f5b7c2b1201d04f35ae4a7a3718d57643d41317ac619ba28f512534 WatchSource:0}: Error finding container 5eeaa6619f5b7c2b1201d04f35ae4a7a3718d57643d41317ac619ba28f512534: Status 404 returned error can't find the container with id 5eeaa6619f5b7c2b1201d04f35ae4a7a3718d57643d41317ac619ba28f512534
Oct 30 05:37:03.489914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:03.489882    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-jbpqr" event={"ID":"945cda97-a3c0-4488-a5d1-5bc921afb74f","Type":"ContainerStarted","Data":"f1e1b701ab3c2d441df34a3248066ffab96556050670c06b8e14c2b1deff15d6"}
Oct 30 05:37:03.490401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:03.490009    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:37:03.491196 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:03.491156    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql" event={"ID":"db89190c-4734-40d6-a4e3-ab0ca9c35cec","Type":"ContainerStarted","Data":"5eeaa6619f5b7c2b1201d04f35ae4a7a3718d57643d41317ac619ba28f512534"}
Oct 30 05:37:03.533687 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:03.533646    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-jbpqr" podStartSLOduration=1.922285362 podStartE2EDuration="4.533634006s" totalImagesPullingTime="2.611348644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:03.533100792 +0000 UTC m=+338.364137999" watchObservedRunningTime="2026-10-30 05:37:03.533634006 +0000 UTC m=+338.364671223"
Oct 30 05:37:06.288334 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:06.288306    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:37:06.288590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:06.288370    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:37:06.288590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:06.288405    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:37:06.506696 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:06.506662    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql" event={"ID":"db89190c-4734-40d6-a4e3-ab0ca9c35cec","Type":"ContainerStarted","Data":"ae11e9f99be899bdb31777174b61c0a322af6cbfa31b08d3830273682ae7582a"}
Oct 30 05:37:06.582396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:06.582303    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql" podStartSLOduration=1.858531682 podStartE2EDuration="5.582288118s" totalImagesPullingTime="3.723756436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:06.568460817 +0000 UTC m=+341.399498034" watchObservedRunningTime="2026-10-30 05:37:06.582288118 +0000 UTC m=+341.413325335"
Oct 30 05:37:07.227412 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:07.227378    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:07.231970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:07.231947    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:07.510987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:07.510914    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:07.511970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:07.511949    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb54lwql"
Oct 30 05:37:07.847819 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:07.847747    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:08.042726 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:08.042698    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:14.499197 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:14.499149    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-jbpqr"
Oct 30 05:37:30.375390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:30.375348    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:32.371918 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:37:32.371882    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:37:56.140745 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.140643    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:56.399263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.399146    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qwdtw"]
Oct 30 05:37:56.402443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.402427    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:37:56.433697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.433674    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 05:37:56.433823 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.433750    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zn6tp\""
Oct 30 05:37:56.434661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.434642    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qwdtw"]
Oct 30 05:37:56.454408 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.454387    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"]
Oct 30 05:37:56.457567 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.457545    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:37:56.489376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.489355    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"]
Oct 30 05:37:56.504271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.504249    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rtcjb"]
Oct 30 05:37:56.507476 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.507459    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:37:56.520557 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.520539    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-k78gk\""
Oct 30 05:37:56.520640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.520620    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 05:37:56.526005 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.525985    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rtcjb"]
Oct 30 05:37:56.548459 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.548430    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5tx\" (UniqueName: \"kubernetes.io/projected/e228b425-837e-4ba9-beb9-17efa76f7b8d-kube-api-access-kr5tx\") pod \"kserve-controller-manager-776cc66d47-qwdtw\" (UID: \"e228b425-837e-4ba9-beb9-17efa76f7b8d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:37:56.548565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.548469    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98758e33-00f8-4172-a9fa-089e41de38dd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bs449\" (UID: \"98758e33-00f8-4172-a9fa-089e41de38dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:37:56.548565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.548502    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e228b425-837e-4ba9-beb9-17efa76f7b8d-cert\") pod \"kserve-controller-manager-776cc66d47-qwdtw\" (UID: \"e228b425-837e-4ba9-beb9-17efa76f7b8d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:37:56.548665 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.548640    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cscz\" (UniqueName: \"kubernetes.io/projected/98758e33-00f8-4172-a9fa-089e41de38dd-kube-api-access-9cscz\") pod \"llmisvc-controller-manager-869d5dcc5d-bs449\" (UID: \"98758e33-00f8-4172-a9fa-089e41de38dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:37:56.562802 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.562779    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-v7p4v"]
Oct 30 05:37:56.565845 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.565831    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.591334 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.591312    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-v7p4v"]
Oct 30 05:37:56.624415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.624380    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 05:37:56.624606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.624507    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kt82w\""
Oct 30 05:37:56.624606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.624533    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 05:37:56.624720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.624641    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 05:37:56.624720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.624714    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2sqs9\""
Oct 30 05:37:56.649714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.649650    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblrl\" (UniqueName: \"kubernetes.io/projected/72779e27-398d-40b1-8c12-c53f7ff376fb-kube-api-access-cblrl\") pod \"model-serving-api-75bd89fdfc-rtcjb\" (UID: \"72779e27-398d-40b1-8c12-c53f7ff376fb\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:37:56.649714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.649690    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5tx\" (UniqueName: \"kubernetes.io/projected/e228b425-837e-4ba9-beb9-17efa76f7b8d-kube-api-access-kr5tx\") pod \"kserve-controller-manager-776cc66d47-qwdtw\" (UID: \"e228b425-837e-4ba9-beb9-17efa76f7b8d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:37:56.649879 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.649711    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98758e33-00f8-4172-a9fa-089e41de38dd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bs449\" (UID: \"98758e33-00f8-4172-a9fa-089e41de38dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:37:56.649939 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.649876    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e228b425-837e-4ba9-beb9-17efa76f7b8d-cert\") pod \"kserve-controller-manager-776cc66d47-qwdtw\" (UID: \"e228b425-837e-4ba9-beb9-17efa76f7b8d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:37:56.649939 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.649913    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72779e27-398d-40b1-8c12-c53f7ff376fb-tls-certs\") pod \"model-serving-api-75bd89fdfc-rtcjb\" (UID: \"72779e27-398d-40b1-8c12-c53f7ff376fb\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:37:56.650031 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.649966    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cscz\" (UniqueName: \"kubernetes.io/projected/98758e33-00f8-4172-a9fa-089e41de38dd-kube-api-access-9cscz\") pod \"llmisvc-controller-manager-869d5dcc5d-bs449\" (UID: \"98758e33-00f8-4172-a9fa-089e41de38dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:37:56.652459 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.652438    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98758e33-00f8-4172-a9fa-089e41de38dd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bs449\" (UID: \"98758e33-00f8-4172-a9fa-089e41de38dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:37:56.652548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.652519    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e228b425-837e-4ba9-beb9-17efa76f7b8d-cert\") pod \"kserve-controller-manager-776cc66d47-qwdtw\" (UID: \"e228b425-837e-4ba9-beb9-17efa76f7b8d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:37:56.674150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.674122    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5tx\" (UniqueName: \"kubernetes.io/projected/e228b425-837e-4ba9-beb9-17efa76f7b8d-kube-api-access-kr5tx\") pod \"kserve-controller-manager-776cc66d47-qwdtw\" (UID: \"e228b425-837e-4ba9-beb9-17efa76f7b8d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:37:56.674255 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.674172    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cscz\" (UniqueName: \"kubernetes.io/projected/98758e33-00f8-4172-a9fa-089e41de38dd-kube-api-access-9cscz\") pod \"llmisvc-controller-manager-869d5dcc5d-bs449\" (UID: \"98758e33-00f8-4172-a9fa-089e41de38dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:37:56.714801 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.714777    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:37:56.750903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.750872    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-metrics-certs\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.751063 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.750940    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswfz\" (UniqueName: \"kubernetes.io/projected/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-kube-api-access-wswfz\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.751133 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.751080    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-cert\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.751215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.751136    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblrl\" (UniqueName: \"kubernetes.io/projected/72779e27-398d-40b1-8c12-c53f7ff376fb-kube-api-access-cblrl\") pod \"model-serving-api-75bd89fdfc-rtcjb\" (UID: \"72779e27-398d-40b1-8c12-c53f7ff376fb\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:37:56.751347 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.751294    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72779e27-398d-40b1-8c12-c53f7ff376fb-tls-certs\") pod \"model-serving-api-75bd89fdfc-rtcjb\" (UID: \"72779e27-398d-40b1-8c12-c53f7ff376fb\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:37:56.754227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.754201    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72779e27-398d-40b1-8c12-c53f7ff376fb-tls-certs\") pod \"model-serving-api-75bd89fdfc-rtcjb\" (UID: \"72779e27-398d-40b1-8c12-c53f7ff376fb\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:37:56.769714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.769691    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:37:56.776617 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.776559    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblrl\" (UniqueName: \"kubernetes.io/projected/72779e27-398d-40b1-8c12-c53f7ff376fb-kube-api-access-cblrl\") pod \"model-serving-api-75bd89fdfc-rtcjb\" (UID: \"72779e27-398d-40b1-8c12-c53f7ff376fb\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:37:56.820426 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.820394    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:37:56.852842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.852801    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-metrics-certs\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.853004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.852885    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswfz\" (UniqueName: \"kubernetes.io/projected/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-kube-api-access-wswfz\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.853004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.852915    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-cert\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.855814 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.855784    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-cert\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.856678 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.856660    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-metrics-certs\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:56.878693 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:56.878667    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswfz\" (UniqueName: \"kubernetes.io/projected/9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66-kube-api-access-wswfz\") pod \"odh-model-controller-f87fcf56c-v7p4v\" (UID: \"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66\") " pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:57.047312 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:37:57.047286    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode228b425_837e_4ba9_beb9_17efa76f7b8d.slice/crio-b9af326161b93bff1c6bedba8d4aced12ee678a4dc1a6cede326e014fcf042ff WatchSource:0}: Error finding container b9af326161b93bff1c6bedba8d4aced12ee678a4dc1a6cede326e014fcf042ff: Status 404 returned error can't find the container with id b9af326161b93bff1c6bedba8d4aced12ee678a4dc1a6cede326e014fcf042ff
Oct 30 05:37:57.055049 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.055025    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qwdtw"]
Oct 30 05:37:57.096778 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.096755    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"]
Oct 30 05:37:57.099105 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:37:57.099079    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98758e33_00f8_4172_a9fa_089e41de38dd.slice/crio-a7f3360eef7228f06f746a3f349de747f18345a5240ddbf7795b3fb4fba2c353 WatchSource:0}: Error finding container a7f3360eef7228f06f746a3f349de747f18345a5240ddbf7795b3fb4fba2c353: Status 404 returned error can't find the container with id a7f3360eef7228f06f746a3f349de747f18345a5240ddbf7795b3fb4fba2c353
Oct 30 05:37:57.138089 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.138063    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rtcjb"]
Oct 30 05:37:57.140857 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:37:57.140829    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72779e27_398d_40b1_8c12_c53f7ff376fb.slice/crio-04c04c95aee5967d253fb0b7ce7e8a24e305140dd8a4b6e41d3e709b25522797 WatchSource:0}: Error finding container 04c04c95aee5967d253fb0b7ce7e8a24e305140dd8a4b6e41d3e709b25522797: Status 404 returned error can't find the container with id 04c04c95aee5967d253fb0b7ce7e8a24e305140dd8a4b6e41d3e709b25522797
Oct 30 05:37:57.178100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.178074    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:37:57.316045 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.315968    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:57.475903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.475779    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-v7p4v"]
Oct 30 05:37:57.478010 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:37:57.477986    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9290a5ba_e7ac_4eaa_89e5_99a9b56f2c66.slice/crio-907a50fdba326b9892d24699ff5ebe07fa68b65975684692cbb6b8934269eb0e WatchSource:0}: Error finding container 907a50fdba326b9892d24699ff5ebe07fa68b65975684692cbb6b8934269eb0e: Status 404 returned error can't find the container with id 907a50fdba326b9892d24699ff5ebe07fa68b65975684692cbb6b8934269eb0e
Oct 30 05:37:57.711561 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.711472    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v" event={"ID":"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66","Type":"ContainerStarted","Data":"907a50fdba326b9892d24699ff5ebe07fa68b65975684692cbb6b8934269eb0e"}
Oct 30 05:37:57.713446 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.713412    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449" event={"ID":"98758e33-00f8-4172-a9fa-089e41de38dd","Type":"ContainerStarted","Data":"a7f3360eef7228f06f746a3f349de747f18345a5240ddbf7795b3fb4fba2c353"}
Oct 30 05:37:57.715140 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.715107    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw" event={"ID":"e228b425-837e-4ba9-beb9-17efa76f7b8d","Type":"ContainerStarted","Data":"b9af326161b93bff1c6bedba8d4aced12ee678a4dc1a6cede326e014fcf042ff"}
Oct 30 05:37:57.717614 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:57.717544    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb" event={"ID":"72779e27-398d-40b1-8c12-c53f7ff376fb","Type":"ContainerStarted","Data":"04c04c95aee5967d253fb0b7ce7e8a24e305140dd8a4b6e41d3e709b25522797"}
Oct 30 05:37:59.082199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:59.082149    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:59.313453 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:59.313417    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:59.467456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:59.467416    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:59.627469 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:37:59.627435    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:00.753064 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:00.753016    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb" event={"ID":"72779e27-398d-40b1-8c12-c53f7ff376fb","Type":"ContainerStarted","Data":"b42e08f2361f3e00644fe1ba563e1aeebe619c3cb13c8295b822ce0f07b10e19"}
Oct 30 05:38:00.753507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:00.753412    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:38:00.797692 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:00.797636    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb" podStartSLOduration=1.875200282 podStartE2EDuration="4.79761857s" totalImagesPullingTime="2.922418288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:00.797525778 +0000 UTC m=+395.628563008" watchObservedRunningTime="2026-10-30 05:38:00.79761857 +0000 UTC m=+395.628655794"
Oct 30 05:38:01.759432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.759342    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw" event={"ID":"e228b425-837e-4ba9-beb9-17efa76f7b8d","Type":"ContainerStarted","Data":"68469f68ce5552747bd203c6c8246fd3177838c8e08d99fb140dde1c38ff7869"}
Oct 30 05:38:01.759978 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.759662    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:38:01.761108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.761083    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v" event={"ID":"9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66","Type":"ContainerStarted","Data":"d572b26e07d98fc38a85ecd7e52ac7c1d6bf1664ec0303ad80df375ca7c3d8ac"}
Oct 30 05:38:01.761262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.761240    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:38:01.763016 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.762985    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449" event={"ID":"98758e33-00f8-4172-a9fa-089e41de38dd","Type":"ContainerStarted","Data":"cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74"}
Oct 30 05:38:01.763174 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.763155    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:38:01.799543 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.799497    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw" podStartSLOduration=1.497357915 podStartE2EDuration="5.799485324s" totalImagesPullingTime="4.302127409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:01.799039404 +0000 UTC m=+396.630076620" watchObservedRunningTime="2026-10-30 05:38:01.799485324 +0000 UTC m=+396.630522541"
Oct 30 05:38:01.839402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.839334    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449" podStartSLOduration=1.739388462 podStartE2EDuration="5.839317435s" totalImagesPullingTime="4.099928973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:01.838559237 +0000 UTC m=+396.669596454" watchObservedRunningTime="2026-10-30 05:38:01.839317435 +0000 UTC m=+396.670354661"
Oct 30 05:38:01.882020 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:01.881943    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v" podStartSLOduration=1.9563128060000001 podStartE2EDuration="5.881925869s" totalImagesPullingTime="3.925613063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:01.880267915 +0000 UTC m=+396.711305132" watchObservedRunningTime="2026-10-30 05:38:01.881925869 +0000 UTC m=+396.712963085"
Oct 30 05:38:02.373020 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:38:02.372981    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:38:08.069047 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:08.068999    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:11.768469 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:11.768443    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-rtcjb"
Oct 30 05:38:12.770636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:12.770598    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-v7p4v"
Oct 30 05:38:15.667347 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:15.667305    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:15.804994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:15.804959    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:30.501865 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:30.501830    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:32.372771 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:38:32.372738    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:38:32.771389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:32.771362    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:38:32.774381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:32.774361    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qwdtw"
Oct 30 05:38:46.604762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:46.604722    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:46.767637 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:46.767602    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:50.407922 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.407894    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"]
Oct 30 05:38:50.412173 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.412151    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.440813 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.440790    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 05:38:50.440922 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.440816    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 05:38:50.440922 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.440819    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-9zbd8\""
Oct 30 05:38:50.455274 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.455248    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"]
Oct 30 05:38:50.469061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.469029    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt7bq\" (UniqueName: \"kubernetes.io/projected/04922c1a-7e1d-4167-a92a-579b86b6ee17-kube-api-access-wt7bq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.469233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.469167    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.469287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.469261    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.570553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.570518    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.570730 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.570565    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.570730 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.570603    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt7bq\" (UniqueName: \"kubernetes.io/projected/04922c1a-7e1d-4167-a92a-579b86b6ee17-kube-api-access-wt7bq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.571029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.570954    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.571029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.570982    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.575771 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.575750    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"]
Oct 30 05:38:50.578724 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.578709    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.596369 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.596339    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ltcrl\""
Oct 30 05:38:50.598836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.598796    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt7bq\" (UniqueName: \"kubernetes.io/projected/04922c1a-7e1d-4167-a92a-579b86b6ee17-kube-api-access-wt7bq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.600933 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.600911    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"]
Oct 30 05:38:50.671553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.671476    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.671553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.671509    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.671553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.671538    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxsk\" (UniqueName: \"kubernetes.io/projected/66508c14-a96f-46b7-8778-1016132d46e7-kube-api-access-8vxsk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.685596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.685566    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"]
Oct 30 05:38:50.688873 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.688857    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.714542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.714516    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"]
Oct 30 05:38:50.722097 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.722067    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-qln4l\""
Oct 30 05:38:50.726233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.726214    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:50.773264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.773209    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.773342 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.773257    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.773342 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.773302    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.773441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.773336    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxsk\" (UniqueName: \"kubernetes.io/projected/66508c14-a96f-46b7-8778-1016132d46e7-kube-api-access-8vxsk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.773441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.773385    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zzk\" (UniqueName: \"kubernetes.io/projected/5e469d8d-101b-40de-b002-c1ef8b79d9b0-kube-api-access-j7zzk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.773441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.773425    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.773892 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.773870    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.774093 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.774070    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.809052 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.809015    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxsk\" (UniqueName: \"kubernetes.io/projected/66508c14-a96f-46b7-8778-1016132d46e7-kube-api-access-8vxsk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.852864 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.852832    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"]
Oct 30 05:38:50.857108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.857093    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:50.873969 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.873947    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.874080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.873987    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zzk\" (UniqueName: \"kubernetes.io/projected/5e469d8d-101b-40de-b002-c1ef8b79d9b0-kube-api-access-j7zzk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.874080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.874011    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.874279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.874261    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.874324 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.874305    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.889332 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.889307    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5mx9s\""
Oct 30 05:38:50.894005 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.893984    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:50.903253 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.903165    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"]
Oct 30 05:38:50.929150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.929065    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zzk\" (UniqueName: \"kubernetes.io/projected/5e469d8d-101b-40de-b002-c1ef8b79d9b0-kube-api-access-j7zzk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:50.974690 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.974665    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:50.974844 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.974698    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:50.974844 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:50.974717    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lkh\" (UniqueName: \"kubernetes.io/projected/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-kube-api-access-z8lkh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:51.004801 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.004776    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:51.035351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.035324    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"]
Oct 30 05:38:51.038538 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:38:51.038513    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04922c1a_7e1d_4167_a92a_579b86b6ee17.slice/crio-85938eadd231a779e6a7bdfb947307da2d5d1533d0c2949cc06f6dcafe5a30be WatchSource:0}: Error finding container 85938eadd231a779e6a7bdfb947307da2d5d1533d0c2949cc06f6dcafe5a30be: Status 404 returned error can't find the container with id 85938eadd231a779e6a7bdfb947307da2d5d1533d0c2949cc06f6dcafe5a30be
Oct 30 05:38:51.076918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.076223    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:51.076918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.076256    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:51.076918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.076277    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lkh\" (UniqueName: \"kubernetes.io/projected/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-kube-api-access-z8lkh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:51.076918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.076871    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:51.077159 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.077128    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:51.107134 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.107105    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lkh\" (UniqueName: \"kubernetes.io/projected/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-kube-api-access-z8lkh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:51.172901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.172873    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:51.194407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.194349    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"]
Oct 30 05:38:51.199495 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:38:51.199469    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66508c14_a96f_46b7_8778_1016132d46e7.slice/crio-c04564c04886d745d09eeef6ebcf51272b0d1c7497fae66977aa3caf588a5822 WatchSource:0}: Error finding container c04564c04886d745d09eeef6ebcf51272b0d1c7497fae66977aa3caf588a5822: Status 404 returned error can't find the container with id c04564c04886d745d09eeef6ebcf51272b0d1c7497fae66977aa3caf588a5822
Oct 30 05:38:51.313881 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.313859    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"]
Oct 30 05:38:51.352114 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:38:51.352078    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e469d8d_101b_40de_b002_c1ef8b79d9b0.slice/crio-9c4b620a509155c5b11c5571dbc5ff0f479592c4b6043970ceb1abb35f6cacb3 WatchSource:0}: Error finding container 9c4b620a509155c5b11c5571dbc5ff0f479592c4b6043970ceb1abb35f6cacb3: Status 404 returned error can't find the container with id 9c4b620a509155c5b11c5571dbc5ff0f479592c4b6043970ceb1abb35f6cacb3
Oct 30 05:38:51.474616 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.474594    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"]
Oct 30 05:38:51.506056 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:38:51.506013    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6eb22c8_9f25_4a33_99a6_1ca20a498c9b.slice/crio-7fbb458d4445739f66a5370db7e17dbc4d5a18de51c792efd38687cda4e13018 WatchSource:0}: Error finding container 7fbb458d4445739f66a5370db7e17dbc4d5a18de51c792efd38687cda4e13018: Status 404 returned error can't find the container with id 7fbb458d4445739f66a5370db7e17dbc4d5a18de51c792efd38687cda4e13018
Oct 30 05:38:51.988856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.988823    5250 generic.go:356] "Generic (PLEG): container finished" podID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerID="c455cb537e3ca7dae1a67e1048d49b2f9fc86bddd8b240001ea8656e57a43dd2" exitCode=0
Oct 30 05:38:51.989025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.988907    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw" event={"ID":"04922c1a-7e1d-4167-a92a-579b86b6ee17","Type":"ContainerDied","Data":"c455cb537e3ca7dae1a67e1048d49b2f9fc86bddd8b240001ea8656e57a43dd2"}
Oct 30 05:38:51.989025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.988967    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw" event={"ID":"04922c1a-7e1d-4167-a92a-579b86b6ee17","Type":"ContainerStarted","Data":"85938eadd231a779e6a7bdfb947307da2d5d1533d0c2949cc06f6dcafe5a30be"}
Oct 30 05:38:51.990659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.990629    5250 generic.go:356] "Generic (PLEG): container finished" podID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerID="ccd4309b85999e667135c5ead8fd08df55b8b7a90d7052bbed2992c2c474b9cb" exitCode=0
Oct 30 05:38:51.990757 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.990699    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm" event={"ID":"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b","Type":"ContainerDied","Data":"ccd4309b85999e667135c5ead8fd08df55b8b7a90d7052bbed2992c2c474b9cb"}
Oct 30 05:38:51.990757 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.990749    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm" event={"ID":"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b","Type":"ContainerStarted","Data":"7fbb458d4445739f66a5370db7e17dbc4d5a18de51c792efd38687cda4e13018"}
Oct 30 05:38:51.992445 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.992415    5250 generic.go:356] "Generic (PLEG): container finished" podID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerID="86eb0d0a85fdbd0ed157ac23549316391658d0ba526459ba93e100f1e4107dfa" exitCode=0
Oct 30 05:38:51.992579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.992494    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6" event={"ID":"5e469d8d-101b-40de-b002-c1ef8b79d9b0","Type":"ContainerDied","Data":"86eb0d0a85fdbd0ed157ac23549316391658d0ba526459ba93e100f1e4107dfa"}
Oct 30 05:38:51.992579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.992527    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6" event={"ID":"5e469d8d-101b-40de-b002-c1ef8b79d9b0","Type":"ContainerStarted","Data":"9c4b620a509155c5b11c5571dbc5ff0f479592c4b6043970ceb1abb35f6cacb3"}
Oct 30 05:38:51.994224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.994119    5250 generic.go:356] "Generic (PLEG): container finished" podID="66508c14-a96f-46b7-8778-1016132d46e7" containerID="53559904174ca40a72fc5b75e970f52c38cfd553b9b9aee26a112ba0099612cf" exitCode=0
Oct 30 05:38:51.994224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.994167    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg" event={"ID":"66508c14-a96f-46b7-8778-1016132d46e7","Type":"ContainerDied","Data":"53559904174ca40a72fc5b75e970f52c38cfd553b9b9aee26a112ba0099612cf"}
Oct 30 05:38:51.994329 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:51.994235    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg" event={"ID":"66508c14-a96f-46b7-8778-1016132d46e7","Type":"ContainerStarted","Data":"c04564c04886d745d09eeef6ebcf51272b0d1c7497fae66977aa3caf588a5822"}
Oct 30 05:38:53.000094 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:52.999999    5250 generic.go:356] "Generic (PLEG): container finished" podID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerID="129c8281db3c812452a72fc152099ff9e88727d2cce96e7a972b0de9541c6ea6" exitCode=0
Oct 30 05:38:53.000094 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:53.000075    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm" event={"ID":"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b","Type":"ContainerDied","Data":"129c8281db3c812452a72fc152099ff9e88727d2cce96e7a972b0de9541c6ea6"}
Oct 30 05:38:53.002590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:53.002567    5250 generic.go:356] "Generic (PLEG): container finished" podID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerID="0de43ac25384fe9d97b701f139e70f9979d500d049863751b32960aa4b40a7ad" exitCode=0
Oct 30 05:38:53.002704 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:53.002644    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6" event={"ID":"5e469d8d-101b-40de-b002-c1ef8b79d9b0","Type":"ContainerDied","Data":"0de43ac25384fe9d97b701f139e70f9979d500d049863751b32960aa4b40a7ad"}
Oct 30 05:38:53.004541 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:53.004520    5250 generic.go:356] "Generic (PLEG): container finished" podID="66508c14-a96f-46b7-8778-1016132d46e7" containerID="7ac816c49ebba1bcf47747484080dd761703094eaf2c9cd047f6be4cf4d10047" exitCode=0
Oct 30 05:38:53.004629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:53.004603    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg" event={"ID":"66508c14-a96f-46b7-8778-1016132d46e7","Type":"ContainerDied","Data":"7ac816c49ebba1bcf47747484080dd761703094eaf2c9cd047f6be4cf4d10047"}
Oct 30 05:38:53.006621 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:53.006600    5250 generic.go:356] "Generic (PLEG): container finished" podID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerID="483a563528c7fd1a769d09bbc7c857ba2a19293ca6cbce4eaa3eac1324645c03" exitCode=0
Oct 30 05:38:53.006702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:53.006645    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw" event={"ID":"04922c1a-7e1d-4167-a92a-579b86b6ee17","Type":"ContainerDied","Data":"483a563528c7fd1a769d09bbc7c857ba2a19293ca6cbce4eaa3eac1324645c03"}
Oct 30 05:38:54.013850 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:54.013750    5250 generic.go:356] "Generic (PLEG): container finished" podID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerID="be2faf5399d763b123c43052d553b3a596861200505e1991233d6bb755f5ed25" exitCode=0
Oct 30 05:38:54.014312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:54.013848    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw" event={"ID":"04922c1a-7e1d-4167-a92a-579b86b6ee17","Type":"ContainerDied","Data":"be2faf5399d763b123c43052d553b3a596861200505e1991233d6bb755f5ed25"}
Oct 30 05:38:54.016036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:54.016015    5250 generic.go:356] "Generic (PLEG): container finished" podID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerID="db630d577b38113b3ee6b05f75e88744f9e8668d6bab378788d1174c54c8ac4d" exitCode=0
Oct 30 05:38:54.016135 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:54.016082    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm" event={"ID":"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b","Type":"ContainerDied","Data":"db630d577b38113b3ee6b05f75e88744f9e8668d6bab378788d1174c54c8ac4d"}
Oct 30 05:38:54.018056 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:54.018035    5250 generic.go:356] "Generic (PLEG): container finished" podID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerID="eb21446a05f113b3d0725b42626f838aa6b95d3ff869c232cb3ffc038ad299a7" exitCode=0
Oct 30 05:38:54.018162 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:54.018100    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6" event={"ID":"5e469d8d-101b-40de-b002-c1ef8b79d9b0","Type":"ContainerDied","Data":"eb21446a05f113b3d0725b42626f838aa6b95d3ff869c232cb3ffc038ad299a7"}
Oct 30 05:38:54.020054 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:54.020034    5250 generic.go:356] "Generic (PLEG): container finished" podID="66508c14-a96f-46b7-8778-1016132d46e7" containerID="9fffbb9baf363283ffe01db81886317adb300a2893033ec662f8cef5bdd6502b" exitCode=0
Oct 30 05:38:54.020133 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:54.020083    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg" event={"ID":"66508c14-a96f-46b7-8778-1016132d46e7","Type":"ContainerDied","Data":"9fffbb9baf363283ffe01db81886317adb300a2893033ec662f8cef5bdd6502b"}
Oct 30 05:38:55.201480 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.201455    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:55.204664 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.204641    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:55.232529 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.232507    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:55.235495 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.235478    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:55.314779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.314713    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66508c14-a96f-46b7-8778-1016132d46e7-kube-api-access-8vxsk\" (UniqueName: \"kubernetes.io/projected/66508c14-a96f-46b7-8778-1016132d46e7-kube-api-access-8vxsk\") pod \"66508c14-a96f-46b7-8778-1016132d46e7\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") "
Oct 30 05:38:55.314899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.314798    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-bundle\") pod \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") "
Oct 30 05:38:55.314899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.314840    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04922c1a-7e1d-4167-a92a-579b86b6ee17-kube-api-access-wt7bq\" (UniqueName: \"kubernetes.io/projected/04922c1a-7e1d-4167-a92a-579b86b6ee17-kube-api-access-wt7bq\") pod \"04922c1a-7e1d-4167-a92a-579b86b6ee17\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") "
Oct 30 05:38:55.314899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.314869    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-util\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-util\") pod \"04922c1a-7e1d-4167-a92a-579b86b6ee17\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") "
Oct 30 05:38:55.314899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.314887    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-kube-api-access-z8lkh\" (UniqueName: \"kubernetes.io/projected/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-kube-api-access-z8lkh\") pod \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") "
Oct 30 05:38:55.315113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.314911    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-bundle\") pod \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") "
Oct 30 05:38:55.315113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.314944    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-bundle\") pod \"04922c1a-7e1d-4167-a92a-579b86b6ee17\" (UID: \"04922c1a-7e1d-4167-a92a-579b86b6ee17\") "
Oct 30 05:38:55.315113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.314970    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e469d8d-101b-40de-b002-c1ef8b79d9b0-kube-api-access-j7zzk\" (UniqueName: \"kubernetes.io/projected/5e469d8d-101b-40de-b002-c1ef8b79d9b0-kube-api-access-j7zzk\") pod \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") "
Oct 30 05:38:55.315113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.315001    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-bundle\") pod \"66508c14-a96f-46b7-8778-1016132d46e7\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") "
Oct 30 05:38:55.315113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.315026    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-util\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-util\") pod \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\" (UID: \"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b\") "
Oct 30 05:38:55.315113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.315059    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-util\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-util\") pod \"66508c14-a96f-46b7-8778-1016132d46e7\" (UID: \"66508c14-a96f-46b7-8778-1016132d46e7\") "
Oct 30 05:38:55.315448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.315127    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-util\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-util\") pod \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\" (UID: \"5e469d8d-101b-40de-b002-c1ef8b79d9b0\") "
Oct 30 05:38:55.315666 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.315638    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-bundle" pod "5e469d8d-101b-40de-b002-c1ef8b79d9b0" (UID: "5e469d8d-101b-40de-b002-c1ef8b79d9b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:55.315964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.315927    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-bundle" pod "b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" (UID: "b6eb22c8-9f25-4a33-99a6-1ca20a498c9b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:55.317805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.317772    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-kube-api-access-z8lkh" pod "b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" (UID: "b6eb22c8-9f25-4a33-99a6-1ca20a498c9b"). InnerVolumeSpecName "kube-api-access-z8lkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:55.318047 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.318020    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66508c14-a96f-46b7-8778-1016132d46e7-kube-api-access-8vxsk" pod "66508c14-a96f-46b7-8778-1016132d46e7" (UID: "66508c14-a96f-46b7-8778-1016132d46e7"). InnerVolumeSpecName "kube-api-access-8vxsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:55.318149 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.318080    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-bundle" pod "04922c1a-7e1d-4167-a92a-579b86b6ee17" (UID: "04922c1a-7e1d-4167-a92a-579b86b6ee17"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:55.318566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.318538    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-bundle" pod "66508c14-a96f-46b7-8778-1016132d46e7" (UID: "66508c14-a96f-46b7-8778-1016132d46e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:55.318718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.318687    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e469d8d-101b-40de-b002-c1ef8b79d9b0-kube-api-access-j7zzk" pod "5e469d8d-101b-40de-b002-c1ef8b79d9b0" (UID: "5e469d8d-101b-40de-b002-c1ef8b79d9b0"). InnerVolumeSpecName "kube-api-access-j7zzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:55.318816 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.318799    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04922c1a-7e1d-4167-a92a-579b86b6ee17-kube-api-access-wt7bq" pod "04922c1a-7e1d-4167-a92a-579b86b6ee17" (UID: "04922c1a-7e1d-4167-a92a-579b86b6ee17"). InnerVolumeSpecName "kube-api-access-wt7bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:55.323164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.323140    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-util" pod "5e469d8d-101b-40de-b002-c1ef8b79d9b0" (UID: "5e469d8d-101b-40de-b002-c1ef8b79d9b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:55.324904 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.324882    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-util" pod "b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" (UID: "b6eb22c8-9f25-4a33-99a6-1ca20a498c9b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:55.325043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.324914    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-util" pod "04922c1a-7e1d-4167-a92a-579b86b6ee17" (UID: "04922c1a-7e1d-4167-a92a-579b86b6ee17"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:55.326072 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.325991    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-util" pod "66508c14-a96f-46b7-8778-1016132d46e7" (UID: "66508c14-a96f-46b7-8778-1016132d46e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:55.416661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416627    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxsk\" (UniqueName: \"kubernetes.io/projected/66508c14-a96f-46b7-8778-1016132d46e7-kube-api-access-8vxsk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416653    5250 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-bundle\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416667    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt7bq\" (UniqueName: \"kubernetes.io/projected/04922c1a-7e1d-4167-a92a-579b86b6ee17-kube-api-access-wt7bq\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416681    5250 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-util\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416694    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8lkh\" (UniqueName: \"kubernetes.io/projected/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-kube-api-access-z8lkh\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416705    5250 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-bundle\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416716    5250 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04922c1a-7e1d-4167-a92a-579b86b6ee17-bundle\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416727    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7zzk\" (UniqueName: \"kubernetes.io/projected/5e469d8d-101b-40de-b002-c1ef8b79d9b0-kube-api-access-j7zzk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416739    5250 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-bundle\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416750    5250 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6eb22c8-9f25-4a33-99a6-1ca20a498c9b-util\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416761    5250 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66508c14-a96f-46b7-8778-1016132d46e7-util\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:55.416899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:55.416771    5250 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e469d8d-101b-40de-b002-c1ef8b79d9b0-util\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:38:56.031055 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.031026    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw"
Oct 30 05:38:56.031275 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.031054    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x2lhw" event={"ID":"04922c1a-7e1d-4167-a92a-579b86b6ee17","Type":"ContainerDied","Data":"85938eadd231a779e6a7bdfb947307da2d5d1533d0c2949cc06f6dcafe5a30be"}
Oct 30 05:38:56.031275 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.031112    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85938eadd231a779e6a7bdfb947307da2d5d1533d0c2949cc06f6dcafe5a30be"
Oct 30 05:38:56.032996 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.032974    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm"
Oct 30 05:38:56.033128 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.032968    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvjhm" event={"ID":"b6eb22c8-9f25-4a33-99a6-1ca20a498c9b","Type":"ContainerDied","Data":"7fbb458d4445739f66a5370db7e17dbc4d5a18de51c792efd38687cda4e13018"}
Oct 30 05:38:56.033128 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.033088    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fbb458d4445739f66a5370db7e17dbc4d5a18de51c792efd38687cda4e13018"
Oct 30 05:38:56.034937 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.034917    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6" event={"ID":"5e469d8d-101b-40de-b002-c1ef8b79d9b0","Type":"ContainerDied","Data":"9c4b620a509155c5b11c5571dbc5ff0f479592c4b6043970ceb1abb35f6cacb3"}
Oct 30 05:38:56.035053 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.034945    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs66v6"
Oct 30 05:38:56.035053 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.034957    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c4b620a509155c5b11c5571dbc5ff0f479592c4b6043970ceb1abb35f6cacb3"
Oct 30 05:38:56.044778 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.044750    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg" event={"ID":"66508c14-a96f-46b7-8778-1016132d46e7","Type":"ContainerDied","Data":"c04564c04886d745d09eeef6ebcf51272b0d1c7497fae66977aa3caf588a5822"}
Oct 30 05:38:56.045029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.044878    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c04564c04886d745d09eeef6ebcf51272b0d1c7497fae66977aa3caf588a5822"
Oct 30 05:38:56.045347 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:56.045335    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajjxpg"
Oct 30 05:38:59.611868 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:38:59.611833    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:02.386234 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:39:02.386162    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:39:02.439261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:02.439218    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:04.293856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:04.293825    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:04.483254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:04.483222    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:05.017982 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:05.017951    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:05.413763 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:05.413722    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:05.691076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:05.690988    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:06.620432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:06.620398    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:06.878436 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:06.878353    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:08.641603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:08.641571    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:09.012515 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:09.012436    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:10.543792 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:10.543756    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:10.728524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:10.728489    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:12.367786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:12.367750    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:12.593612 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:12.593580    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:15.990305 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990269    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"]
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990586    5250 cpu_manager.go:414] "containerMap: removing container" podUID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerName="pull"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990596    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerName="pull"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990611    5250 cpu_manager.go:414] "containerMap: removing container" podUID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerName="extract"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990615    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerName="extract"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990624    5250 cpu_manager.go:414] "containerMap: removing container" podUID="66508c14-a96f-46b7-8778-1016132d46e7" containerName="pull"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990629    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66508c14-a96f-46b7-8778-1016132d46e7" containerName="pull"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990637    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerName="extract"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990641    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerName="extract"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990649    5250 cpu_manager.go:414] "containerMap: removing container" podUID="66508c14-a96f-46b7-8778-1016132d46e7" containerName="util"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990653    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66508c14-a96f-46b7-8778-1016132d46e7" containerName="util"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990658    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerName="pull"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990662    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerName="pull"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990669    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerName="pull"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990672    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerName="pull"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990680    5250 cpu_manager.go:414] "containerMap: removing container" podUID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerName="util"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990684    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerName="util"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990689    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerName="util"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990693    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerName="util"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990705    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerName="util"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990709    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerName="util"
Oct 30 05:39:15.990710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990719    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerName="extract"
Oct 30 05:39:15.991399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990724    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerName="extract"
Oct 30 05:39:15.991399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990731    5250 cpu_manager.go:414] "containerMap: removing container" podUID="66508c14-a96f-46b7-8778-1016132d46e7" containerName="extract"
Oct 30 05:39:15.991399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990735    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66508c14-a96f-46b7-8778-1016132d46e7" containerName="extract"
Oct 30 05:39:15.991399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990780    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="04922c1a-7e1d-4167-a92a-579b86b6ee17" containerName="extract"
Oct 30 05:39:15.991399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990788    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6eb22c8-9f25-4a33-99a6-1ca20a498c9b" containerName="extract"
Oct 30 05:39:15.991399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990797    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e469d8d-101b-40de-b002-c1ef8b79d9b0" containerName="extract"
Oct 30 05:39:15.991399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.990802    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="66508c14-a96f-46b7-8778-1016132d46e7" containerName="extract"
Oct 30 05:39:15.993830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:15.993811    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.020948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.020928    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 05:39:16.032757 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.032732    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pljfh\""
Oct 30 05:39:16.032893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.032799    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 05:39:16.032893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.032830    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 05:39:16.032893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.032840    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 05:39:16.034448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.034425    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"]
Oct 30 05:39:16.060790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.060760    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:16.094503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.094479    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85d307eb-8102-41eb-9eb2-4197a4d4683e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.094609 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.094506    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvpd\" (UniqueName: \"kubernetes.io/projected/85d307eb-8102-41eb-9eb2-4197a4d4683e-kube-api-access-nnvpd\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.094609 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.094528    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85d307eb-8102-41eb-9eb2-4197a4d4683e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.195780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.195750    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85d307eb-8102-41eb-9eb2-4197a4d4683e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.195940 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.195849    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85d307eb-8102-41eb-9eb2-4197a4d4683e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.195940 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.195877    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvpd\" (UniqueName: \"kubernetes.io/projected/85d307eb-8102-41eb-9eb2-4197a4d4683e-kube-api-access-nnvpd\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.196530 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.196509    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85d307eb-8102-41eb-9eb2-4197a4d4683e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.198413 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.198397    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85d307eb-8102-41eb-9eb2-4197a4d4683e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.218698 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.218674    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvpd\" (UniqueName: \"kubernetes.io/projected/85d307eb-8102-41eb-9eb2-4197a4d4683e-kube-api-access-nnvpd\") pod \"kuadrant-console-plugin-66c4fb575d-x7mw6\" (UID: \"85d307eb-8102-41eb-9eb2-4197a4d4683e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.223338 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.223314    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:16.306244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.306139    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"
Oct 30 05:39:16.617658 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:16.617586    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6"]
Oct 30 05:39:16.620879 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:39:16.620853    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85d307eb_8102_41eb_9eb2_4197a4d4683e.slice/crio-a410ce8fb29c8af04dbe3f63916c6c67c3c1dbc9d6bddbc35bb449cb1280f8f8 WatchSource:0}: Error finding container a410ce8fb29c8af04dbe3f63916c6c67c3c1dbc9d6bddbc35bb449cb1280f8f8: Status 404 returned error can't find the container with id a410ce8fb29c8af04dbe3f63916c6c67c3c1dbc9d6bddbc35bb449cb1280f8f8
Oct 30 05:39:17.147075 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:17.147032    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6" event={"ID":"85d307eb-8102-41eb-9eb2-4197a4d4683e","Type":"ContainerStarted","Data":"a410ce8fb29c8af04dbe3f63916c6c67c3c1dbc9d6bddbc35bb449cb1280f8f8"}
Oct 30 05:39:20.125401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:20.125363    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:20.278011 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:20.277971    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:22.174164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:22.174127    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6" event={"ID":"85d307eb-8102-41eb-9eb2-4197a4d4683e","Type":"ContainerStarted","Data":"8bd19515bb35574d248ef30f09fe504928e5e7f099c9367ce60de5d38f08b1f4"}
Oct 30 05:39:22.217500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:22.217453    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7mw6" podStartSLOduration=2.73857716 podStartE2EDuration="7.217437144s" totalImagesPullingTime="4.478859984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:22.217207029 +0000 UTC m=+477.048244245" watchObservedRunningTime="2026-10-30 05:39:22.217437144 +0000 UTC m=+477.048474357"
Oct 30 05:39:26.139308 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.139274    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"]
Oct 30 05:39:26.143746 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.143729    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:26.157550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.157527    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"]
Oct 30 05:39:26.167702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.167679    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2lbtb\""
Oct 30 05:39:26.196323 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.196284    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-s7df4\" (UID: \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:26.196323 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.196332    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnb8v\" (UniqueName: \"kubernetes.io/projected/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-kube-api-access-lnb8v\") pod \"kuadrant-operator-controller-manager-956956bc4-s7df4\" (UID: \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:26.297080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.297045    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-s7df4\" (UID: \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:26.297080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.297087    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnb8v\" (UniqueName: \"kubernetes.io/projected/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-kube-api-access-lnb8v\") pod \"kuadrant-operator-controller-manager-956956bc4-s7df4\" (UID: \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:26.297556 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.297534    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-s7df4\" (UID: \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:26.324418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.324383    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnb8v\" (UniqueName: \"kubernetes.io/projected/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-kube-api-access-lnb8v\") pod \"kuadrant-operator-controller-manager-956956bc4-s7df4\" (UID: \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:26.456911 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.456841    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:26.798836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:26.798653    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"]
Oct 30 05:39:26.801653 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:39:26.801626    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279ec430_d4c1_494c_b9ce_03fe6cc93a0f.slice/crio-8e7ddedfc6051c6d765a94b1f4f5efd1a7e388e06212e337dc9da75a61b72904 WatchSource:0}: Error finding container 8e7ddedfc6051c6d765a94b1f4f5efd1a7e388e06212e337dc9da75a61b72904: Status 404 returned error can't find the container with id 8e7ddedfc6051c6d765a94b1f4f5efd1a7e388e06212e337dc9da75a61b72904
Oct 30 05:39:27.202827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.202791    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4" event={"ID":"279ec430-d4c1-494c-b9ce-03fe6cc93a0f","Type":"ContainerStarted","Data":"8e7ddedfc6051c6d765a94b1f4f5efd1a7e388e06212e337dc9da75a61b72904"}
Oct 30 05:39:27.707976 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.707944    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"]
Oct 30 05:39:27.719912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.719879    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"]
Oct 30 05:39:27.735782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.735757    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"]
Oct 30 05:39:27.740691 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.740671    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:27.768717 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.768682    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"]
Oct 30 05:39:27.809173 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.809143    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05026cd0-f858-420b-a15e-b2f9cc916bcd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-gcq97\" (UID: \"05026cd0-f858-420b-a15e-b2f9cc916bcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:27.809360 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.809209    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5f4\" (UniqueName: \"kubernetes.io/projected/05026cd0-f858-420b-a15e-b2f9cc916bcd-kube-api-access-lm5f4\") pod \"kuadrant-operator-controller-manager-956956bc4-gcq97\" (UID: \"05026cd0-f858-420b-a15e-b2f9cc916bcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:27.910610 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.910578    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05026cd0-f858-420b-a15e-b2f9cc916bcd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-gcq97\" (UID: \"05026cd0-f858-420b-a15e-b2f9cc916bcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:27.910783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.910617    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5f4\" (UniqueName: \"kubernetes.io/projected/05026cd0-f858-420b-a15e-b2f9cc916bcd-kube-api-access-lm5f4\") pod \"kuadrant-operator-controller-manager-956956bc4-gcq97\" (UID: \"05026cd0-f858-420b-a15e-b2f9cc916bcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:27.910969 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.910946    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05026cd0-f858-420b-a15e-b2f9cc916bcd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-gcq97\" (UID: \"05026cd0-f858-420b-a15e-b2f9cc916bcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:27.958893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:27.958813    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5f4\" (UniqueName: \"kubernetes.io/projected/05026cd0-f858-420b-a15e-b2f9cc916bcd-kube-api-access-lm5f4\") pod \"kuadrant-operator-controller-manager-956956bc4-gcq97\" (UID: \"05026cd0-f858-420b-a15e-b2f9cc916bcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:28.059531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:28.059495    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:28.400797 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:28.400771    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"]
Oct 30 05:39:28.416839 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:39:28.416805    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05026cd0_f858_420b_a15e_b2f9cc916bcd.slice/crio-290840ace5a93409231658f8a66c0ec863567e61bc486ef9cb9bd0b02ad64d7c WatchSource:0}: Error finding container 290840ace5a93409231658f8a66c0ec863567e61bc486ef9cb9bd0b02ad64d7c: Status 404 returned error can't find the container with id 290840ace5a93409231658f8a66c0ec863567e61bc486ef9cb9bd0b02ad64d7c
Oct 30 05:39:29.085410 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:29.085371    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:29.215189 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:29.215132    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97" event={"ID":"05026cd0-f858-420b-a15e-b2f9cc916bcd","Type":"ContainerStarted","Data":"290840ace5a93409231658f8a66c0ec863567e61bc486ef9cb9bd0b02ad64d7c"}
Oct 30 05:39:29.492658 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:29.492619    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:30.637610 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:30.637558    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:31.700656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:31.700620    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:31.876036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:31.875998    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:32.379286 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:39:32.379227    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:39:33.241258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:33.241218    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97" event={"ID":"05026cd0-f858-420b-a15e-b2f9cc916bcd","Type":"ContainerStarted","Data":"1e5dee758360f6befc6e382fab4463723db10a354b0a310a7bf1ebda4ccc03a4"}
Oct 30 05:39:33.241702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:33.241286    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:34.791374 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:34.791327    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:34.952432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:34.952402    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:35.267646 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:35.267602    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4" podUID="279ec430-d4c1-494c-b9ce-03fe6cc93a0f" containerName="manager" containerID="cri-o://42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961" gracePeriod=2
Oct 30 05:39:35.513593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:35.513571    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:35.577052 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:35.576983    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-extensions-socket-volume\") pod \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\" (UID: \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\") "
Oct 30 05:39:35.577205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:35.577105    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-kube-api-access-lnb8v\" (UniqueName: \"kubernetes.io/projected/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-kube-api-access-lnb8v\") pod \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\" (UID: \"279ec430-d4c1-494c-b9ce-03fe6cc93a0f\") "
Oct 30 05:39:35.577284 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:35.577263    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-extensions-socket-volume" pod "279ec430-d4c1-494c-b9ce-03fe6cc93a0f" (UID: "279ec430-d4c1-494c-b9ce-03fe6cc93a0f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:35.577409 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:35.577388    5250 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-extensions-socket-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:39:35.579315 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:35.579297    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-kube-api-access-lnb8v" pod "279ec430-d4c1-494c-b9ce-03fe6cc93a0f" (UID: "279ec430-d4c1-494c-b9ce-03fe6cc93a0f"). InnerVolumeSpecName "kube-api-access-lnb8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:35.677886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:35.677861    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnb8v\" (UniqueName: \"kubernetes.io/projected/279ec430-d4c1-494c-b9ce-03fe6cc93a0f-kube-api-access-lnb8v\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:39:36.273294 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:36.273262    5250 generic.go:356] "Generic (PLEG): container finished" podID="279ec430-d4c1-494c-b9ce-03fe6cc93a0f" containerID="42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961" exitCode=2
Oct 30 05:39:36.273759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:36.273319    5250 scope.go:122] "RemoveContainer" containerID="42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961"
Oct 30 05:39:36.273759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:36.273381    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:36.282203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:36.282156    5250 scope.go:122] "RemoveContainer" containerID="42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961"
Oct 30 05:39:36.282477 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:39:36.282455    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961\": container with ID starting with 42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961 not found: ID does not exist" containerID="42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961"
Oct 30 05:39:36.282541 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:36.282488    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961"} err="failed to get container status \"42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961\": rpc error: code = NotFound desc = could not find container \"42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961\": container with ID starting with 42e79e74ee49ce3fade56285af524a457e200e72586f8fbfb78cd5c5502f0961 not found: ID does not exist"
Oct 30 05:39:36.286732 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:39:36.286707    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-s7df4\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="279ec430-d4c1-494c-b9ce-03fe6cc93a0f" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:36.299454 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:39:36.299434    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-s7df4\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="279ec430-d4c1-494c-b9ce-03fe6cc93a0f" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-s7df4"
Oct 30 05:39:37.278504 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:37.278467    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279ec430-d4c1-494c-b9ce-03fe6cc93a0f" path="/var/lib/kubelet/pods/279ec430-d4c1-494c-b9ce-03fe6cc93a0f/volumes"
Oct 30 05:39:39.362474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:39.362440    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:39.514409 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:39.514379    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:39.661438 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:39.661384    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:41.089196 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:41.089149    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:41.241204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:41.231791    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:42.208529 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:42.208496    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:42.678145 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:42.678106    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:42.827350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:42.827316    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:44.249741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:44.249710    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:44.320582 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:44.320524    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97" podStartSLOduration=13.568876106 podStartE2EDuration="17.320508346s" totalImagesPullingTime="3.75163224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:33.322585483 +0000 UTC m=+488.153622702" watchObservedRunningTime="2026-10-30 05:39:44.320508346 +0000 UTC m=+499.151545563"
Oct 30 05:39:48.200579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:48.200548    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:48.397628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:48.397595    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:57.172599 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.172554    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"]
Oct 30 05:39:57.173198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.172847    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97" podUID="05026cd0-f858-420b-a15e-b2f9cc916bcd" containerName="manager" containerID="cri-o://1e5dee758360f6befc6e382fab4463723db10a354b0a310a7bf1ebda4ccc03a4" gracePeriod=10
Oct 30 05:39:57.375954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.375922    5250 generic.go:356] "Generic (PLEG): container finished" podID="05026cd0-f858-420b-a15e-b2f9cc916bcd" containerID="1e5dee758360f6befc6e382fab4463723db10a354b0a310a7bf1ebda4ccc03a4" exitCode=0
Oct 30 05:39:57.376131 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.375985    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97" event={"ID":"05026cd0-f858-420b-a15e-b2f9cc916bcd","Type":"ContainerDied","Data":"1e5dee758360f6befc6e382fab4463723db10a354b0a310a7bf1ebda4ccc03a4"}
Oct 30 05:39:57.424213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.424150    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:57.575901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.575862    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05026cd0-f858-420b-a15e-b2f9cc916bcd-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05026cd0-f858-420b-a15e-b2f9cc916bcd-extensions-socket-volume\") pod \"05026cd0-f858-420b-a15e-b2f9cc916bcd\" (UID: \"05026cd0-f858-420b-a15e-b2f9cc916bcd\") "
Oct 30 05:39:57.576099 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.575953    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05026cd0-f858-420b-a15e-b2f9cc916bcd-kube-api-access-lm5f4\" (UniqueName: \"kubernetes.io/projected/05026cd0-f858-420b-a15e-b2f9cc916bcd-kube-api-access-lm5f4\") pod \"05026cd0-f858-420b-a15e-b2f9cc916bcd\" (UID: \"05026cd0-f858-420b-a15e-b2f9cc916bcd\") "
Oct 30 05:39:57.576388 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.576361    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05026cd0-f858-420b-a15e-b2f9cc916bcd-extensions-socket-volume" pod "05026cd0-f858-420b-a15e-b2f9cc916bcd" (UID: "05026cd0-f858-420b-a15e-b2f9cc916bcd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:57.578263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.578240    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05026cd0-f858-420b-a15e-b2f9cc916bcd-kube-api-access-lm5f4" pod "05026cd0-f858-420b-a15e-b2f9cc916bcd" (UID: "05026cd0-f858-420b-a15e-b2f9cc916bcd"). InnerVolumeSpecName "kube-api-access-lm5f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:57.677456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.677390    5250 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05026cd0-f858-420b-a15e-b2f9cc916bcd-extensions-socket-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:39:57.677456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:57.677413    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm5f4\" (UniqueName: \"kubernetes.io/projected/05026cd0-f858-420b-a15e-b2f9cc916bcd-kube-api-access-lm5f4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:39:58.381445 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:58.381413    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"
Oct 30 05:39:58.381445 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:58.381440    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97" event={"ID":"05026cd0-f858-420b-a15e-b2f9cc916bcd","Type":"ContainerDied","Data":"290840ace5a93409231658f8a66c0ec863567e61bc486ef9cb9bd0b02ad64d7c"}
Oct 30 05:39:58.381974 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:58.381506    5250 scope.go:122] "RemoveContainer" containerID="1e5dee758360f6befc6e382fab4463723db10a354b0a310a7bf1ebda4ccc03a4"
Oct 30 05:39:58.443661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:58.443633    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"]
Oct 30 05:39:58.455346 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:58.455314    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gcq97"]
Oct 30 05:39:59.278340 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:39:59.278312    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05026cd0-f858-420b-a15e-b2f9cc916bcd" path="/var/lib/kubelet/pods/05026cd0-f858-420b-a15e-b2f9cc916bcd/volumes"
Oct 30 05:40:01.587974 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.587945    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"]
Oct 30 05:40:01.588402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.588304    5250 cpu_manager.go:414] "containerMap: removing container" podUID="279ec430-d4c1-494c-b9ce-03fe6cc93a0f" containerName="manager"
Oct 30 05:40:01.588402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.588317    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279ec430-d4c1-494c-b9ce-03fe6cc93a0f" containerName="manager"
Oct 30 05:40:01.588402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.588336    5250 cpu_manager.go:414] "containerMap: removing container" podUID="05026cd0-f858-420b-a15e-b2f9cc916bcd" containerName="manager"
Oct 30 05:40:01.588402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.588342    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05026cd0-f858-420b-a15e-b2f9cc916bcd" containerName="manager"
Oct 30 05:40:01.588402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.588395    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="05026cd0-f858-420b-a15e-b2f9cc916bcd" containerName="manager"
Oct 30 05:40:01.588402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.588406    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="279ec430-d4c1-494c-b9ce-03fe6cc93a0f" containerName="manager"
Oct 30 05:40:01.592701 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.592681    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.608636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.608614    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.608736 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.608646    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.608736 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.608678    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.608838 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.608753    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.608838 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.608781    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.608935 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.608870    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2ksq\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-kube-api-access-r2ksq\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.608989 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.608943    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.608989 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.608975    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea411ab-37fe-4d3e-bada-24d67576140c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.609090 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.609033    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.628337 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.628314    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"]
Oct 30 05:40:01.638160 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.638142    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-77wcn\""
Oct 30 05:40:01.695736 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.695694    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:01.709676 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709648    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.709842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709688    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea411ab-37fe-4d3e-bada-24d67576140c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.709842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709705    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.709842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709745    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.709842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709766    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.709842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709798    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.709842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709834    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.710166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709874    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.710166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.709902    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2ksq\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-kube-api-access-r2ksq\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.710300 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.710202    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.710356 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.710300    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.710356 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.710335    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.710464 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.710394    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.710566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.710544    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea411ab-37fe-4d3e-bada-24d67576140c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.712212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.712172    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.712575 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.712557    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.743358 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.743332    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2ksq\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-kube-api-access-r2ksq\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.743511 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.743335    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-s4njj\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:01.873574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.873478    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:01.905468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:01.905435    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:02.226708 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:02.226670    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"]
Oct 30 05:40:02.231712 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:40:02.231682    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea411ab_37fe_4d3e_bada_24d67576140c.slice/crio-7cd41e4ca2e2c4b4a8b5bc97c76bfccf16ac88962258fc832879e3770e6bd2d0 WatchSource:0}: Error finding container 7cd41e4ca2e2c4b4a8b5bc97c76bfccf16ac88962258fc832879e3770e6bd2d0: Status 404 returned error can't find the container with id 7cd41e4ca2e2c4b4a8b5bc97c76bfccf16ac88962258fc832879e3770e6bd2d0
Oct 30 05:40:02.233854 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:02.233822    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:40:02.233938 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:02.233894    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:40:02.233938 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:02.233921    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:40:02.373546 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:40:02.373500    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:40:02.402727 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:02.402691    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" event={"ID":"6ea411ab-37fe-4d3e-bada-24d67576140c","Type":"ContainerStarted","Data":"90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f"}
Oct 30 05:40:02.402889 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:02.402747    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" event={"ID":"6ea411ab-37fe-4d3e-bada-24d67576140c","Type":"ContainerStarted","Data":"7cd41e4ca2e2c4b4a8b5bc97c76bfccf16ac88962258fc832879e3770e6bd2d0"}
Oct 30 05:40:02.456257 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:02.456196    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" podStartSLOduration=1.456161719 podStartE2EDuration="1.456161719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:02.454712483 +0000 UTC m=+517.285749700" watchObservedRunningTime="2026-10-30 05:40:02.456161719 +0000 UTC m=+517.287198936"
Oct 30 05:40:02.906199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:02.906145    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:03.911639 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:03.911609    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:04.411622 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:04.411582    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:04.412650 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:04.412628    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:40:17.103440 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:17.103393    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:17.271357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:17.271315    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:17.484338 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:17.484306    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:17.966461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:17.966429    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p7qjf"]
Oct 30 05:40:17.969885 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:17.969869    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-p7qjf"
Oct 30 05:40:17.992242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:17.992213    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p7qjf"]
Oct 30 05:40:17.993551 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:17.993529    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pr45g\""
Oct 30 05:40:18.047520 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:18.047494    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rln\" (UniqueName: \"kubernetes.io/projected/9f4d7d92-832a-4dd4-83af-7f38802483ec-kube-api-access-67rln\") pod \"authorino-57f7f8659f-p7qjf\" (UID: \"9f4d7d92-832a-4dd4-83af-7f38802483ec\") " pod="kuadrant-system/authorino-57f7f8659f-p7qjf"
Oct 30 05:40:18.087622 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:18.087585    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:18.148869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:18.148837    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rln\" (UniqueName: \"kubernetes.io/projected/9f4d7d92-832a-4dd4-83af-7f38802483ec-kube-api-access-67rln\") pod \"authorino-57f7f8659f-p7qjf\" (UID: \"9f4d7d92-832a-4dd4-83af-7f38802483ec\") " pod="kuadrant-system/authorino-57f7f8659f-p7qjf"
Oct 30 05:40:18.176263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:18.176237    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rln\" (UniqueName: \"kubernetes.io/projected/9f4d7d92-832a-4dd4-83af-7f38802483ec-kube-api-access-67rln\") pod \"authorino-57f7f8659f-p7qjf\" (UID: \"9f4d7d92-832a-4dd4-83af-7f38802483ec\") " pod="kuadrant-system/authorino-57f7f8659f-p7qjf"
Oct 30 05:40:18.252211 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:18.252130    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:18.281274 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:18.281253    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-p7qjf"
Oct 30 05:40:18.574084 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:18.574053    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p7qjf"]
Oct 30 05:40:18.576732 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:40:18.576705    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f4d7d92_832a_4dd4_83af_7f38802483ec.slice/crio-31b7f377f7eaadbc3f57746ab98a5d9c2a6e535883751598100c3061b707e506 WatchSource:0}: Error finding container 31b7f377f7eaadbc3f57746ab98a5d9c2a6e535883751598100c3061b707e506: Status 404 returned error can't find the container with id 31b7f377f7eaadbc3f57746ab98a5d9c2a6e535883751598100c3061b707e506
Oct 30 05:40:19.483493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:19.483457    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-p7qjf" event={"ID":"9f4d7d92-832a-4dd4-83af-7f38802483ec","Type":"ContainerStarted","Data":"31b7f377f7eaadbc3f57746ab98a5d9c2a6e535883751598100c3061b707e506"}
Oct 30 05:40:20.490534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:20.490497    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-p7qjf" event={"ID":"9f4d7d92-832a-4dd4-83af-7f38802483ec","Type":"ContainerStarted","Data":"45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f"}
Oct 30 05:40:20.527407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:20.527355    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-p7qjf" podStartSLOduration=2.01537496 podStartE2EDuration="3.527339063s" totalImagesPullingTime="1.511964103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:20.527035801 +0000 UTC m=+535.358073020" watchObservedRunningTime="2026-10-30 05:40:20.527339063 +0000 UTC m=+535.358376281"
Oct 30 05:40:21.642615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:21.642575    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p7qjf"]
Oct 30 05:40:22.500606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:22.500539    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-p7qjf" podUID="9f4d7d92-832a-4dd4-83af-7f38802483ec" containerName="authorino" containerID="cri-o://45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f" gracePeriod=30
Oct 30 05:40:22.748303 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:22.748267    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-p7qjf"
Oct 30 05:40:22.790316 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:22.790247    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4d7d92-832a-4dd4-83af-7f38802483ec-kube-api-access-67rln\" (UniqueName: \"kubernetes.io/projected/9f4d7d92-832a-4dd4-83af-7f38802483ec-kube-api-access-67rln\") pod \"9f4d7d92-832a-4dd4-83af-7f38802483ec\" (UID: \"9f4d7d92-832a-4dd4-83af-7f38802483ec\") "
Oct 30 05:40:22.792475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:22.792429    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4d7d92-832a-4dd4-83af-7f38802483ec-kube-api-access-67rln" pod "9f4d7d92-832a-4dd4-83af-7f38802483ec" (UID: "9f4d7d92-832a-4dd4-83af-7f38802483ec"). InnerVolumeSpecName "kube-api-access-67rln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:22.891719 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:22.891682    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rln\" (UniqueName: \"kubernetes.io/projected/9f4d7d92-832a-4dd4-83af-7f38802483ec-kube-api-access-67rln\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:40:23.506441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.506403    5250 generic.go:356] "Generic (PLEG): container finished" podID="9f4d7d92-832a-4dd4-83af-7f38802483ec" containerID="45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f" exitCode=0
Oct 30 05:40:23.506441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.506443    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-p7qjf"
Oct 30 05:40:23.506698 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.506480    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-p7qjf" event={"ID":"9f4d7d92-832a-4dd4-83af-7f38802483ec","Type":"ContainerDied","Data":"45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f"}
Oct 30 05:40:23.506698 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.506519    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-p7qjf" event={"ID":"9f4d7d92-832a-4dd4-83af-7f38802483ec","Type":"ContainerDied","Data":"31b7f377f7eaadbc3f57746ab98a5d9c2a6e535883751598100c3061b707e506"}
Oct 30 05:40:23.506698 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.506540    5250 scope.go:122] "RemoveContainer" containerID="45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f"
Oct 30 05:40:23.514956 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.514938    5250 scope.go:122] "RemoveContainer" containerID="45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f"
Oct 30 05:40:23.515230 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:40:23.515210    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f\": container with ID starting with 45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f not found: ID does not exist" containerID="45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f"
Oct 30 05:40:23.515280 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.515240    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f"} err="failed to get container status \"45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f\": rpc error: code = NotFound desc = could not find container \"45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f\": container with ID starting with 45d7d4a86d3e154e4412e797f7f1c26e8d000f1e1a8f998016dab6b7dd26406f not found: ID does not exist"
Oct 30 05:40:23.556012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.555949    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p7qjf"]
Oct 30 05:40:23.567611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:23.567588    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p7qjf"]
Oct 30 05:40:25.279674 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:25.279643    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4d7d92-832a-4dd4-83af-7f38802483ec" path="/var/lib/kubelet/pods/9f4d7d92-832a-4dd4-83af-7f38802483ec/volumes"
Oct 30 05:40:30.379710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:30.379663    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:30.737383 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:30.737345    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:32.373111 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:40:32.373079    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:40:36.115666 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.115596    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:36.261145 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.261108    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:36.764481 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.764447    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"]
Oct 30 05:40:36.764822 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.764808    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9f4d7d92-832a-4dd4-83af-7f38802483ec" containerName="authorino"
Oct 30 05:40:36.764867 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.764824    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4d7d92-832a-4dd4-83af-7f38802483ec" containerName="authorino"
Oct 30 05:40:36.764922 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.764913    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4d7d92-832a-4dd4-83af-7f38802483ec" containerName="authorino"
Oct 30 05:40:36.767848 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.767830    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:36.786256 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.786228    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"]
Oct 30 05:40:36.796310 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.796264    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:40:36.796310 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.796264    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:40:36.796496 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.796324    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-crt2l\""
Oct 30 05:40:36.796496 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.796386    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 05:40:36.886899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.886870    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:36.913204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.913157    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0b4e236c-65dc-4595-81b7-008a8f71d7e5-data\") pod \"postgres-7cdbf6dc44-c789c\" (UID: \"0b4e236c-65dc-4595-81b7-008a8f71d7e5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:36.913346 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:36.913272    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4hz8\" (UniqueName: \"kubernetes.io/projected/0b4e236c-65dc-4595-81b7-008a8f71d7e5-kube-api-access-p4hz8\") pod \"postgres-7cdbf6dc44-c789c\" (UID: \"0b4e236c-65dc-4595-81b7-008a8f71d7e5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:37.014131 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:37.014094    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0b4e236c-65dc-4595-81b7-008a8f71d7e5-data\") pod \"postgres-7cdbf6dc44-c789c\" (UID: \"0b4e236c-65dc-4595-81b7-008a8f71d7e5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:37.014320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:37.014147    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4hz8\" (UniqueName: \"kubernetes.io/projected/0b4e236c-65dc-4595-81b7-008a8f71d7e5-kube-api-access-p4hz8\") pod \"postgres-7cdbf6dc44-c789c\" (UID: \"0b4e236c-65dc-4595-81b7-008a8f71d7e5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:37.014583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:37.014540    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0b4e236c-65dc-4595-81b7-008a8f71d7e5-data\") pod \"postgres-7cdbf6dc44-c789c\" (UID: \"0b4e236c-65dc-4595-81b7-008a8f71d7e5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:37.045397 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:37.045373    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4hz8\" (UniqueName: \"kubernetes.io/projected/0b4e236c-65dc-4595-81b7-008a8f71d7e5-kube-api-access-p4hz8\") pod \"postgres-7cdbf6dc44-c789c\" (UID: \"0b4e236c-65dc-4595-81b7-008a8f71d7e5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:37.081292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:37.081264    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:37.380074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:37.380018    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"]
Oct 30 05:40:37.382787 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:40:37.382761    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4e236c_65dc_4595_81b7_008a8f71d7e5.slice/crio-9f8c1e95e19eec411639ba110d37d82f60dae386371e46bfa8df5186a1c19398 WatchSource:0}: Error finding container 9f8c1e95e19eec411639ba110d37d82f60dae386371e46bfa8df5186a1c19398: Status 404 returned error can't find the container with id 9f8c1e95e19eec411639ba110d37d82f60dae386371e46bfa8df5186a1c19398
Oct 30 05:40:37.384071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:37.384045    5250 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:40:37.576341 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:37.576306    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c" event={"ID":"0b4e236c-65dc-4595-81b7-008a8f71d7e5","Type":"ContainerStarted","Data":"9f8c1e95e19eec411639ba110d37d82f60dae386371e46bfa8df5186a1c19398"}
Oct 30 05:40:42.606634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:42.606555    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c" event={"ID":"0b4e236c-65dc-4595-81b7-008a8f71d7e5","Type":"ContainerStarted","Data":"260e7e303ed712f2a942044024a356e03406baf3a6361e4758e46a12cec58bcf"}
Oct 30 05:40:42.607015 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:42.606745    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:42.647564 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:42.647518    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c" podStartSLOduration=1.728319207 podStartE2EDuration="6.647504744s" totalImagesPullingTime="4.919185537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:42.647005361 +0000 UTC m=+557.478042771" watchObservedRunningTime="2026-10-30 05:40:42.647504744 +0000 UTC m=+557.478541961"
Oct 30 05:40:48.640894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:48.640862    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c789c"
Oct 30 05:40:49.165702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.165653    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:49.312228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.312195    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:49.440400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.440318    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzhcm"]
Oct 30 05:40:49.445487 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.445465    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hzhcm"
Oct 30 05:40:49.461779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.461756    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzhcm"]
Oct 30 05:40:49.471448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.471430    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pr45g\""
Oct 30 05:40:49.631154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.631127    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lj8\" (UniqueName: \"kubernetes.io/projected/9b63790a-1110-4048-941f-8881150f7d38-kube-api-access-v7lj8\") pod \"authorino-6f6944bcdd-hzhcm\" (UID: \"9b63790a-1110-4048-941f-8881150f7d38\") " pod="kuadrant-system/authorino-6f6944bcdd-hzhcm"
Oct 30 05:40:49.677137 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.677106    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzhcm"]
Oct 30 05:40:49.677508 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:40:49.677321    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-v7lj8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-hzhcm" podUID="9b63790a-1110-4048-941f-8881150f7d38"
Oct 30 05:40:49.731752 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.731660    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lj8\" (UniqueName: \"kubernetes.io/projected/9b63790a-1110-4048-941f-8881150f7d38-kube-api-access-v7lj8\") pod \"authorino-6f6944bcdd-hzhcm\" (UID: \"9b63790a-1110-4048-941f-8881150f7d38\") " pod="kuadrant-system/authorino-6f6944bcdd-hzhcm"
Oct 30 05:40:49.755097 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.755075    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lj8\" (UniqueName: \"kubernetes.io/projected/9b63790a-1110-4048-941f-8881150f7d38-kube-api-access-v7lj8\") pod \"authorino-6f6944bcdd-hzhcm\" (UID: \"9b63790a-1110-4048-941f-8881150f7d38\") " pod="kuadrant-system/authorino-6f6944bcdd-hzhcm"
Oct 30 05:40:49.846027 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.845993    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66ff994f45-ljpk2"]
Oct 30 05:40:49.849727 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.849712    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:40:49.868484 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.868458    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 05:40:49.869711 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:49.869691    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66ff994f45-ljpk2"]
Oct 30 05:40:50.035294 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.035212    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba383a08-51da-4ebd-a46e-57ecdac36b23-tls-cert\") pod \"authorino-66ff994f45-ljpk2\" (UID: \"ba383a08-51da-4ebd-a46e-57ecdac36b23\") " pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:40:50.035294 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.035253    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cm7s\" (UniqueName: \"kubernetes.io/projected/ba383a08-51da-4ebd-a46e-57ecdac36b23-kube-api-access-6cm7s\") pod \"authorino-66ff994f45-ljpk2\" (UID: \"ba383a08-51da-4ebd-a46e-57ecdac36b23\") " pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:40:50.135966 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.135934    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cm7s\" (UniqueName: \"kubernetes.io/projected/ba383a08-51da-4ebd-a46e-57ecdac36b23-kube-api-access-6cm7s\") pod \"authorino-66ff994f45-ljpk2\" (UID: \"ba383a08-51da-4ebd-a46e-57ecdac36b23\") " pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:40:50.136120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.136009    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba383a08-51da-4ebd-a46e-57ecdac36b23-tls-cert\") pod \"authorino-66ff994f45-ljpk2\" (UID: \"ba383a08-51da-4ebd-a46e-57ecdac36b23\") " pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:40:50.138531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.138499    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba383a08-51da-4ebd-a46e-57ecdac36b23-tls-cert\") pod \"authorino-66ff994f45-ljpk2\" (UID: \"ba383a08-51da-4ebd-a46e-57ecdac36b23\") " pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:40:50.159456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.159436    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cm7s\" (UniqueName: \"kubernetes.io/projected/ba383a08-51da-4ebd-a46e-57ecdac36b23-kube-api-access-6cm7s\") pod \"authorino-66ff994f45-ljpk2\" (UID: \"ba383a08-51da-4ebd-a46e-57ecdac36b23\") " pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:40:50.161263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.161242    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:40:50.459060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.459022    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66ff994f45-ljpk2"]
Oct 30 05:40:50.461914 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:40:50.461887    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba383a08_51da_4ebd_a46e_57ecdac36b23.slice/crio-21095a7c9b0526550c73744b48fb90baa87d14bb128a321667f1339978fae1d2 WatchSource:0}: Error finding container 21095a7c9b0526550c73744b48fb90baa87d14bb128a321667f1339978fae1d2: Status 404 returned error can't find the container with id 21095a7c9b0526550c73744b48fb90baa87d14bb128a321667f1339978fae1d2
Oct 30 05:40:50.645155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.645121    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66ff994f45-ljpk2" event={"ID":"ba383a08-51da-4ebd-a46e-57ecdac36b23","Type":"ContainerStarted","Data":"21095a7c9b0526550c73744b48fb90baa87d14bb128a321667f1339978fae1d2"}
Oct 30 05:40:50.645155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.645145    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hzhcm"
Oct 30 05:40:50.653367 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.653343    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hzhcm"
Oct 30 05:40:50.740743 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.740668    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b63790a-1110-4048-941f-8881150f7d38-kube-api-access-v7lj8\" (UniqueName: \"kubernetes.io/projected/9b63790a-1110-4048-941f-8881150f7d38-kube-api-access-v7lj8\") pod \"9b63790a-1110-4048-941f-8881150f7d38\" (UID: \"9b63790a-1110-4048-941f-8881150f7d38\") "
Oct 30 05:40:50.742994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.742974    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b63790a-1110-4048-941f-8881150f7d38-kube-api-access-v7lj8" pod "9b63790a-1110-4048-941f-8881150f7d38" (UID: "9b63790a-1110-4048-941f-8881150f7d38"). InnerVolumeSpecName "kube-api-access-v7lj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:50.841142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:50.841113    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7lj8\" (UniqueName: \"kubernetes.io/projected/9b63790a-1110-4048-941f-8881150f7d38-kube-api-access-v7lj8\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:40:51.651596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:51.651561    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66ff994f45-ljpk2" event={"ID":"ba383a08-51da-4ebd-a46e-57ecdac36b23","Type":"ContainerStarted","Data":"840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9"}
Oct 30 05:40:51.651783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:51.651610    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hzhcm"
Oct 30 05:40:51.706374 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:51.706328    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-66ff994f45-ljpk2" podStartSLOduration=2.115022655 podStartE2EDuration="2.706313553s" totalImagesPullingTime="591.290898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:51.704467694 +0000 UTC m=+566.535504912" watchObservedRunningTime="2026-10-30 05:40:51.706313553 +0000 UTC m=+566.537350782"
Oct 30 05:40:51.771569 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:51.771543    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzhcm"]
Oct 30 05:40:51.774995 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:51.774972    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzhcm"]
Oct 30 05:40:53.288213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:53.288159    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b63790a-1110-4048-941f-8881150f7d38" path="/var/lib/kubelet/pods/9b63790a-1110-4048-941f-8881150f7d38/volumes"
Oct 30 05:40:59.231379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:59.231335    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:59.473219 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:59.473164    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:59.625920 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:59.625843    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:59.770856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:40:59.770826    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:02.371968 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:41:02.371926    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:41:02.796587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:02.796555    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:02.961329 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:02.961297    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:06.393537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:06.393505    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:06.625941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:06.625902    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:07.675619 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:07.675584    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:07.952026 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:07.951936    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:09.153467 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:09.153437    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:09.325043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:09.325010    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:25.334052 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:25.334024    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:41:25.334841 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:25.334824    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:41:30.897261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:30.897212    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:32.372866 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:41:32.372830    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:41:32.428407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:32.428363    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:32.598496 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:32.598466    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:33.841663 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:33.841630    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:33.986074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:33.986045    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:34.154470 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:34.154442    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:44.695214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.695111    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"]
Oct 30 05:41:44.697549 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.697530    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:44.714467 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.714440    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/961f8fab-46d9-484a-99e4-a0b80a50ef08-cert\") pod \"llmisvc-controller-manager-6466c554d8-4nqxz\" (UID: \"961f8fab-46d9-484a-99e4-a0b80a50ef08\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:44.714597 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.714494    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hr7h\" (UniqueName: \"kubernetes.io/projected/961f8fab-46d9-484a-99e4-a0b80a50ef08-kube-api-access-9hr7h\") pod \"llmisvc-controller-manager-6466c554d8-4nqxz\" (UID: \"961f8fab-46d9-484a-99e4-a0b80a50ef08\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:44.717964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.717941    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"]
Oct 30 05:41:44.815487 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.815448    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/961f8fab-46d9-484a-99e4-a0b80a50ef08-cert\") pod \"llmisvc-controller-manager-6466c554d8-4nqxz\" (UID: \"961f8fab-46d9-484a-99e4-a0b80a50ef08\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:44.815669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.815527    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hr7h\" (UniqueName: \"kubernetes.io/projected/961f8fab-46d9-484a-99e4-a0b80a50ef08-kube-api-access-9hr7h\") pod \"llmisvc-controller-manager-6466c554d8-4nqxz\" (UID: \"961f8fab-46d9-484a-99e4-a0b80a50ef08\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:44.818038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.818019    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/961f8fab-46d9-484a-99e4-a0b80a50ef08-cert\") pod \"llmisvc-controller-manager-6466c554d8-4nqxz\" (UID: \"961f8fab-46d9-484a-99e4-a0b80a50ef08\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:44.838651 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.838619    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hr7h\" (UniqueName: \"kubernetes.io/projected/961f8fab-46d9-484a-99e4-a0b80a50ef08-kube-api-access-9hr7h\") pod \"llmisvc-controller-manager-6466c554d8-4nqxz\" (UID: \"961f8fab-46d9-484a-99e4-a0b80a50ef08\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:44.913862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.913832    5250 generic.go:356] "Generic (PLEG): container finished" podID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerID="90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f" exitCode=0
Oct 30 05:41:44.914029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.913912    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" event={"ID":"6ea411ab-37fe-4d3e-bada-24d67576140c","Type":"ContainerDied","Data":"90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f"}
Oct 30 05:41:44.914366 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.914347    5250 scope.go:122] "RemoveContainer" containerID="90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f"
Oct 30 05:41:44.915466 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.915438    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:41:44.915558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.915503    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:41:44.915558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:44.915534    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:41:45.009070 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:45.009040    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:45.321906 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:45.321885    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"]
Oct 30 05:41:45.323977 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:41:45.323942    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961f8fab_46d9_484a_99e4_a0b80a50ef08.slice/crio-c30b0ec36bd26db5c2878442d0542d62f9e729471e83e243f886dced73726849 WatchSource:0}: Error finding container c30b0ec36bd26db5c2878442d0542d62f9e729471e83e243f886dced73726849: Status 404 returned error can't find the container with id c30b0ec36bd26db5c2878442d0542d62f9e729471e83e243f886dced73726849
Oct 30 05:41:45.922785 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:45.922685    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" event={"ID":"6ea411ab-37fe-4d3e-bada-24d67576140c","Type":"ContainerStarted","Data":"9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42"}
Oct 30 05:41:45.924492 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:45.924464    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz" event={"ID":"961f8fab-46d9-484a-99e4-a0b80a50ef08","Type":"ContainerStarted","Data":"4d55e1de71359b9fd360e2605539d273096b480ac7b67e30e853f1e04e18f66a"}
Oct 30 05:41:45.924616 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:45.924509    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz" event={"ID":"961f8fab-46d9-484a-99e4-a0b80a50ef08","Type":"ContainerStarted","Data":"c30b0ec36bd26db5c2878442d0542d62f9e729471e83e243f886dced73726849"}
Oct 30 05:41:45.924616 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:45.924580    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:41:46.028465 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:46.028416    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz" podStartSLOduration=1.699238357 podStartE2EDuration="2.028403109s" totalImagesPullingTime="329.164752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:46.027737048 +0000 UTC m=+620.858774265" watchObservedRunningTime="2026-10-30 05:41:46.028403109 +0000 UTC m=+620.859440325"
Oct 30 05:41:46.906023 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:46.905990    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:41:46.907202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:46.907151    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-s4njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 05:41:46.907325 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:46.907226    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 05:41:47.906653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:47.906612    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-s4njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 05:41:47.907168 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:47.906669    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 05:41:48.906456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:48.906416    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-s4njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 05:41:48.906629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:48.906470    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 05:41:49.411840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:49.411811    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:41:49.906713 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:49.906682    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-s4njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 05:41:49.906873 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:49.906738    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 05:41:50.913920 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:50.913881    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:41:50.915052 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:41:50.915032    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 05:42:01.092550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:01.092508    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:02.372416 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:42:02.372384    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:42:06.666569 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:06.666484    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:16.930255 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:16.930224    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4nqxz"
Oct 30 05:42:17.000344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.000298    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"]
Oct 30 05:42:17.000560 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.000538    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449" podUID="98758e33-00f8-4172-a9fa-089e41de38dd" containerName="manager" containerID="cri-o://cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74" gracePeriod=30
Oct 30 05:42:17.247761 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.247739    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:42:17.403750 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.403721    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98758e33-00f8-4172-a9fa-089e41de38dd-kube-api-access-9cscz\" (UniqueName: \"kubernetes.io/projected/98758e33-00f8-4172-a9fa-089e41de38dd-kube-api-access-9cscz\") pod \"98758e33-00f8-4172-a9fa-089e41de38dd\" (UID: \"98758e33-00f8-4172-a9fa-089e41de38dd\") "
Oct 30 05:42:17.403902 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.403764    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98758e33-00f8-4172-a9fa-089e41de38dd-cert\" (UniqueName: \"kubernetes.io/secret/98758e33-00f8-4172-a9fa-089e41de38dd-cert\") pod \"98758e33-00f8-4172-a9fa-089e41de38dd\" (UID: \"98758e33-00f8-4172-a9fa-089e41de38dd\") "
Oct 30 05:42:17.406195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.406146    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98758e33-00f8-4172-a9fa-089e41de38dd-cert" pod "98758e33-00f8-4172-a9fa-089e41de38dd" (UID: "98758e33-00f8-4172-a9fa-089e41de38dd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:42:17.406195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.406171    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98758e33-00f8-4172-a9fa-089e41de38dd-kube-api-access-9cscz" pod "98758e33-00f8-4172-a9fa-089e41de38dd" (UID: "98758e33-00f8-4172-a9fa-089e41de38dd"). InnerVolumeSpecName "kube-api-access-9cscz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:42:17.504638 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.504574    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cscz\" (UniqueName: \"kubernetes.io/projected/98758e33-00f8-4172-a9fa-089e41de38dd-kube-api-access-9cscz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:42:17.504638 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.504598    5250 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98758e33-00f8-4172-a9fa-089e41de38dd-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:42:17.916552 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:17.916517    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:18.082004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.081971    5250 generic.go:356] "Generic (PLEG): container finished" podID="98758e33-00f8-4172-a9fa-089e41de38dd" containerID="cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74" exitCode=0
Oct 30 05:42:18.082485 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.082020    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"
Oct 30 05:42:18.082485 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.082057    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449" event={"ID":"98758e33-00f8-4172-a9fa-089e41de38dd","Type":"ContainerDied","Data":"cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74"}
Oct 30 05:42:18.082485 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.082115    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449" event={"ID":"98758e33-00f8-4172-a9fa-089e41de38dd","Type":"ContainerDied","Data":"a7f3360eef7228f06f746a3f349de747f18345a5240ddbf7795b3fb4fba2c353"}
Oct 30 05:42:18.082485 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.082146    5250 scope.go:122] "RemoveContainer" containerID="cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74"
Oct 30 05:42:18.091357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.091343    5250 scope.go:122] "RemoveContainer" containerID="cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74"
Oct 30 05:42:18.091611 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:42:18.091594    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74\": container with ID starting with cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74 not found: ID does not exist" containerID="cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74"
Oct 30 05:42:18.091681 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.091619    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74"} err="failed to get container status \"cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74\": rpc error: code = NotFound desc = could not find container \"cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74\": container with ID starting with cac9b67e94105e1641089c7d10f19fd6ff5c943928c64843a8ae5376f202ad74 not found: ID does not exist"
Oct 30 05:42:18.164043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.164014    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"]
Oct 30 05:42:18.175327 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.175266    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bs449"]
Oct 30 05:42:18.184167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:18.184144    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:19.278299 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:19.278265    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98758e33-00f8-4172-a9fa-089e41de38dd" path="/var/lib/kubelet/pods/98758e33-00f8-4172-a9fa-089e41de38dd/volumes"
Oct 30 05:42:30.995195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:30.995149    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:32.373238 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:42:32.373201    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:42:34.971355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:34.971317    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:36.491916 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:36.491877    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:39.005167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:39.005130    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:40.622274 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:40.622171    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:41.943462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:41.943426    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:43.084745 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:43.084691    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:45.525163 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:45.525124    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:59.375339 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:42:59.375302    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:00.314149 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.314114    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66ff994f45-ljpk2"]
Oct 30 05:43:00.314333 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.314318    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-66ff994f45-ljpk2" podUID="ba383a08-51da-4ebd-a46e-57ecdac36b23" containerName="authorino" containerID="cri-o://840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9" gracePeriod=30
Oct 30 05:43:00.556805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.556781    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:43:00.654984 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.654938    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:00.668098 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.668078    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba383a08-51da-4ebd-a46e-57ecdac36b23-tls-cert\" (UniqueName: \"kubernetes.io/secret/ba383a08-51da-4ebd-a46e-57ecdac36b23-tls-cert\") pod \"ba383a08-51da-4ebd-a46e-57ecdac36b23\" (UID: \"ba383a08-51da-4ebd-a46e-57ecdac36b23\") "
Oct 30 05:43:00.668244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.668120    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba383a08-51da-4ebd-a46e-57ecdac36b23-kube-api-access-6cm7s\" (UniqueName: \"kubernetes.io/projected/ba383a08-51da-4ebd-a46e-57ecdac36b23-kube-api-access-6cm7s\") pod \"ba383a08-51da-4ebd-a46e-57ecdac36b23\" (UID: \"ba383a08-51da-4ebd-a46e-57ecdac36b23\") "
Oct 30 05:43:00.670196 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.670158    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba383a08-51da-4ebd-a46e-57ecdac36b23-kube-api-access-6cm7s" pod "ba383a08-51da-4ebd-a46e-57ecdac36b23" (UID: "ba383a08-51da-4ebd-a46e-57ecdac36b23"). InnerVolumeSpecName "kube-api-access-6cm7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:43:00.678205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.678167    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba383a08-51da-4ebd-a46e-57ecdac36b23-tls-cert" pod "ba383a08-51da-4ebd-a46e-57ecdac36b23" (UID: "ba383a08-51da-4ebd-a46e-57ecdac36b23"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:43:00.768856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.768828    5250 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba383a08-51da-4ebd-a46e-57ecdac36b23-tls-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:43:00.768856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:00.768854    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cm7s\" (UniqueName: \"kubernetes.io/projected/ba383a08-51da-4ebd-a46e-57ecdac36b23-kube-api-access-6cm7s\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:43:01.304399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.304368    5250 generic.go:356] "Generic (PLEG): container finished" podID="ba383a08-51da-4ebd-a46e-57ecdac36b23" containerID="840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9" exitCode=0
Oct 30 05:43:01.304540 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.304406    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66ff994f45-ljpk2"
Oct 30 05:43:01.304540 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.304435    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66ff994f45-ljpk2" event={"ID":"ba383a08-51da-4ebd-a46e-57ecdac36b23","Type":"ContainerDied","Data":"840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9"}
Oct 30 05:43:01.304540 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.304466    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66ff994f45-ljpk2" event={"ID":"ba383a08-51da-4ebd-a46e-57ecdac36b23","Type":"ContainerDied","Data":"21095a7c9b0526550c73744b48fb90baa87d14bb128a321667f1339978fae1d2"}
Oct 30 05:43:01.304540 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.304482    5250 scope.go:122] "RemoveContainer" containerID="840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9"
Oct 30 05:43:01.313037 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.313017    5250 scope.go:122] "RemoveContainer" containerID="840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9"
Oct 30 05:43:01.313327 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:43:01.313308    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9\": container with ID starting with 840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9 not found: ID does not exist" containerID="840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9"
Oct 30 05:43:01.313383 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.313336    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9"} err="failed to get container status \"840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9\": rpc error: code = NotFound desc = could not find container \"840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9\": container with ID starting with 840084d63e0057a569713f8761b1ff59919ff7383e3ffb8d4effd6a4b35fd9d9 not found: ID does not exist"
Oct 30 05:43:01.356591 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.356561    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66ff994f45-ljpk2"]
Oct 30 05:43:01.367988 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:01.367957    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66ff994f45-ljpk2"]
Oct 30 05:43:02.372669 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:43:02.372632    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:43:03.279143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:03.279099    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba383a08-51da-4ebd-a46e-57ecdac36b23" path="/var/lib/kubelet/pods/ba383a08-51da-4ebd-a46e-57ecdac36b23/volumes"
Oct 30 05:43:04.029409 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:04.029376    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:04.182723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:04.182690    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:10.523235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:10.523204    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:10.663971 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:10.663939    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:17.776531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.776491    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"]
Oct 30 05:43:17.777006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.776960    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ba383a08-51da-4ebd-a46e-57ecdac36b23" containerName="authorino"
Oct 30 05:43:17.777006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.776977    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba383a08-51da-4ebd-a46e-57ecdac36b23" containerName="authorino"
Oct 30 05:43:17.777006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.776996    5250 cpu_manager.go:414] "containerMap: removing container" podUID="98758e33-00f8-4172-a9fa-089e41de38dd" containerName="manager"
Oct 30 05:43:17.777006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.777004    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98758e33-00f8-4172-a9fa-089e41de38dd" containerName="manager"
Oct 30 05:43:17.777244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.777111    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba383a08-51da-4ebd-a46e-57ecdac36b23" containerName="authorino"
Oct 30 05:43:17.777244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.777124    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="98758e33-00f8-4172-a9fa-089e41de38dd" containerName="manager"
Oct 30 05:43:17.783253 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.781716    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.809985 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.809957    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"]
Oct 30 05:43:17.916460 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916424    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.916460 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916465    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.916659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916481    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f236a47e-f79b-4857-9ab1-9711f03815da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.916659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916541    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f236a47e-f79b-4857-9ab1-9711f03815da-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.916659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916602    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.916659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916621    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.916659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916636    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.916817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916662    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvx4\" (UniqueName: \"kubernetes.io/projected/f236a47e-f79b-4857-9ab1-9711f03815da-kube-api-access-9lvx4\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:17.916817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:17.916726    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f236a47e-f79b-4857-9ab1-9711f03815da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017316    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017355    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017369    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f236a47e-f79b-4857-9ab1-9711f03815da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017384    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f236a47e-f79b-4857-9ab1-9711f03815da-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017476    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017517    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017540    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017564    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvx4\" (UniqueName: \"kubernetes.io/projected/f236a47e-f79b-4857-9ab1-9711f03815da-kube-api-access-9lvx4\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017611    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f236a47e-f79b-4857-9ab1-9711f03815da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017786    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017833    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.017982 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.017895    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.018077 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.018055    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.018308 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.018293    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f236a47e-f79b-4857-9ab1-9711f03815da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.019832 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.019815    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f236a47e-f79b-4857-9ab1-9711f03815da-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.019967 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.019950    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f236a47e-f79b-4857-9ab1-9711f03815da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.040216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.040140    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f236a47e-f79b-4857-9ab1-9711f03815da-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.040388 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.040366    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvx4\" (UniqueName: \"kubernetes.io/projected/f236a47e-f79b-4857-9ab1-9711f03815da-kube-api-access-9lvx4\") pod \"maas-default-gateway-openshift-default-84587756dc-6qzpc\" (UID: \"f236a47e-f79b-4857-9ab1-9711f03815da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.096221 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.096172    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:18.411832 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.411811    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"]
Oct 30 05:43:18.413821 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:43:18.413794    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf236a47e_f79b_4857_9ab1_9711f03815da.slice/crio-da8690c2940c2b96e26ab21d416e29fc987aaa4988202e00c70ddf81375d9344 WatchSource:0}: Error finding container da8690c2940c2b96e26ab21d416e29fc987aaa4988202e00c70ddf81375d9344: Status 404 returned error can't find the container with id da8690c2940c2b96e26ab21d416e29fc987aaa4988202e00c70ddf81375d9344
Oct 30 05:43:18.415852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.415819    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:43:18.415942 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.415893    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:43:18.415990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:18.415939    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:43:19.388508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:19.388474    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" event={"ID":"f236a47e-f79b-4857-9ab1-9711f03815da","Type":"ContainerStarted","Data":"45d67fa4b5c5d476928cb4c58f5d346315edf2aac3a450c783e9d165c790bbad"}
Oct 30 05:43:19.389006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:19.388525    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" event={"ID":"f236a47e-f79b-4857-9ab1-9711f03815da","Type":"ContainerStarted","Data":"da8690c2940c2b96e26ab21d416e29fc987aaa4988202e00c70ddf81375d9344"}
Oct 30 05:43:19.479203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:19.478855    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" podStartSLOduration=2.478832325 podStartE2EDuration="2.478832325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:19.443869522 +0000 UTC m=+714.274906739" watchObservedRunningTime="2026-10-30 05:43:19.478832325 +0000 UTC m=+714.309869543"
Oct 30 05:43:20.097258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:20.097231    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:20.098615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:20.098594    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6qzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 05:43:20.098664 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:20.098644    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" podUID="f236a47e-f79b-4857-9ab1-9711f03815da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 05:43:21.097513 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:21.097472    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6qzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 05:43:21.097934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:21.097535    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" podUID="f236a47e-f79b-4857-9ab1-9711f03815da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 05:43:22.096747 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:22.096708    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6qzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 05:43:22.096950 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:22.096792    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" podUID="f236a47e-f79b-4857-9ab1-9711f03815da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 05:43:23.096887 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:23.096846    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6qzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 05:43:23.097330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:23.096904    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" podUID="f236a47e-f79b-4857-9ab1-9711f03815da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 05:43:24.096923 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:24.096886    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6qzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 05:43:24.097358 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:24.096954    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" podUID="f236a47e-f79b-4857-9ab1-9711f03815da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 05:43:25.096630 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:25.096595    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6qzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 05:43:25.096809 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:25.096648    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" podUID="f236a47e-f79b-4857-9ab1-9711f03815da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 05:43:26.608780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:26.608745    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:26.609267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:26.609159    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:26.609949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:26.609929    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc"
Oct 30 05:43:27.371169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:27.371136    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:30.162398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:30.162362    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:30.676363 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:30.676331    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:31.160678 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:31.160645    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:32.345104 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:32.345072    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:32.372957 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:43:32.372924    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:43:32.505981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:32.505950    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:33.981910 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:33.981876    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:34.118679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:43:34.118645    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:00.312161 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:00.312086    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:00.485299 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:00.485268    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:00.865141 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:00.865110    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:01.091640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:01.091603    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:01.302499 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:01.302467    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:01.568597 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:01.568514    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:01.761941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:01.761902    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:01.923408 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:01.923372    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"]
Oct 30 05:44:01.927172 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:01.927150    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:01.943302 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:01.943281    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a14091-openshift-default-dockercfg-xdwgh\""
Oct 30 05:44:01.957372 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:01.957342    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"]
Oct 30 05:44:02.002678 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.002649    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:02.003806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.003786    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.003868 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.003823    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.003868 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.003845    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nlp9\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-kube-api-access-6nlp9\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.004007 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.003985    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.004043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.004017    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.004078 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.004038    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.004122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.004105    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.004156 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.004137    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.004261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.004241    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105289    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105324    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105580 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105342    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105580 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105361    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105580 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105390    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105580 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105423    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105580 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105485    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105580 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105520    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105580 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105548    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlp9\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-kube-api-access-6nlp9\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105813    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105885    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.105934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.105925    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.106198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.106161    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.106432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.106411    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.108461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.108437    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.109017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.108996    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.129166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.129137    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.134983 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.134946    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nlp9\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-kube-api-access-6nlp9\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.178575 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.178470    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:02.241009 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.240976    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:02.356311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.356257    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:02.372165 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:44:02.372117    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:44:02.569406 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.569376    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"]
Oct 30 05:44:02.571059 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:02.571023    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae41e05_46f8_418b_ab84_968a0e5a23b1.slice/crio-ce6664de4234c1db6193c3da81309b11220850b81bba90b1ae5a7581aad686c2 WatchSource:0}: Error finding container ce6664de4234c1db6193c3da81309b11220850b81bba90b1ae5a7581aad686c2: Status 404 returned error can't find the container with id ce6664de4234c1db6193c3da81309b11220850b81bba90b1ae5a7581aad686c2
Oct 30 05:44:02.574053 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.573979    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:44:02.574289 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.574071    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:44:02.574289 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.574111    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:44:02.586057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:02.586023    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" event={"ID":"8ae41e05-46f8-418b-ab84-968a0e5a23b1","Type":"ContainerStarted","Data":"ce6664de4234c1db6193c3da81309b11220850b81bba90b1ae5a7581aad686c2"}
Oct 30 05:44:03.593229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:03.593199    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" event={"ID":"8ae41e05-46f8-418b-ab84-968a0e5a23b1","Type":"ContainerStarted","Data":"0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e"}
Oct 30 05:44:03.666160 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:03.666109    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podStartSLOduration=2.6660938119999997 podStartE2EDuration="2.666093812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:03.664821091 +0000 UTC m=+758.495858307" watchObservedRunningTime="2026-10-30 05:44:03.666093812 +0000 UTC m=+758.497131029"
Oct 30 05:44:04.241633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:04.241597    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:04.246544 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:04.246518    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:04.597660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:04.597582    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:04.598565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:04.598545    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:05.072604 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:05.072571    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:05.245916 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:05.245881    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:05.435801 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:05.435766    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:05.683016 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:05.682981    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:05.892225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:05.892172    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:06.109442 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:06.109413    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:06.914996 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:06.914951    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:07.210573 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:07.210475    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:07.410596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:07.410564    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:08.919191 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:08.919146    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:09.108404 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:09.108320    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:09.365750 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:09.365666    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:09.518307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:09.518277    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:09.720357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:09.720321    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:09.882682 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:09.882651    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:09.989549 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:09.989473    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"]
Oct 30 05:44:09.992834 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:09.992818    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:10.018579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.018554    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"]
Oct 30 05:44:10.027684 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.027665    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-64123f-dockercfg-7r455\""
Oct 30 05:44:10.071818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.071793    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-64123f\""
Oct 30 05:44:10.072069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.072052    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f53076d-3434-4341-bd0c-836f77c610c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4\" (UID: \"9f53076d-3434-4341-bd0c-836f77c610c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:10.072136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.072088    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrztc\" (UniqueName: \"kubernetes.io/projected/9f53076d-3434-4341-bd0c-836f77c610c7-kube-api-access-rrztc\") pod \"payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4\" (UID: \"9f53076d-3434-4341-bd0c-836f77c610c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:10.094887 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.094825    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:10.172790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.172746    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f53076d-3434-4341-bd0c-836f77c610c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4\" (UID: \"9f53076d-3434-4341-bd0c-836f77c610c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:10.172970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.172810    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrztc\" (UniqueName: \"kubernetes.io/projected/9f53076d-3434-4341-bd0c-836f77c610c7-kube-api-access-rrztc\") pod \"payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4\" (UID: \"9f53076d-3434-4341-bd0c-836f77c610c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:10.173576 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.173557    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f53076d-3434-4341-bd0c-836f77c610c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4\" (UID: \"9f53076d-3434-4341-bd0c-836f77c610c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:10.196762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.196727    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrztc\" (UniqueName: \"kubernetes.io/projected/9f53076d-3434-4341-bd0c-836f77c610c7-kube-api-access-rrztc\") pod \"payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4\" (UID: \"9f53076d-3434-4341-bd0c-836f77c610c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:10.306539 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.306459    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:10.610925 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.610888    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"]
Oct 30 05:44:10.614950 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:10.614810    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f53076d_3434_4341_bd0c_836f77c610c7.slice/crio-bd2b4dc1f032b5e2593a0680242f4ecc7ae1bfb171c73454d3a3a799f74a39ec WatchSource:0}: Error finding container bd2b4dc1f032b5e2593a0680242f4ecc7ae1bfb171c73454d3a3a799f74a39ec: Status 404 returned error can't find the container with id bd2b4dc1f032b5e2593a0680242f4ecc7ae1bfb171c73454d3a3a799f74a39ec
Oct 30 05:44:10.628502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:10.628474    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4" event={"ID":"9f53076d-3434-4341-bd0c-836f77c610c7","Type":"ContainerStarted","Data":"bd2b4dc1f032b5e2593a0680242f4ecc7ae1bfb171c73454d3a3a799f74a39ec"}
Oct 30 05:44:11.393700 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:11.393664    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:12.337459 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.337428    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:12.489808 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.489763    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"]
Oct 30 05:44:12.501233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.501210    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:12.514758 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.514733    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"]
Oct 30 05:44:12.531763 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.531734    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:12.552225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.552193    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-be78f2\""
Oct 30 05:44:12.553569 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.553534    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-be78f2-dockercfg-khvsp\""
Oct 30 05:44:12.596066 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.595974    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5rwg\" (UniqueName: \"kubernetes.io/projected/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-kube-api-access-s5rwg\") pod \"payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8\" (UID: \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:12.596238 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.596148    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8\" (UID: \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:12.697591 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.697557    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8\" (UID: \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:12.697751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.697611    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5rwg\" (UniqueName: \"kubernetes.io/projected/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-kube-api-access-s5rwg\") pod \"payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8\" (UID: \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:12.698282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.698261    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8\" (UID: \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:12.726089 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.726056    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5rwg\" (UniqueName: \"kubernetes.io/projected/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-kube-api-access-s5rwg\") pod \"payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8\" (UID: \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:12.817924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:12.817893    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:13.124669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.124646    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"]
Oct 30 05:44:13.127111 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:13.127079    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40cfd76_bd78_423e_8ad7_ef3fee34eb52.slice/crio-5a45e3b3e1b9c33e7f1145ad728f4870c9c3874e4640591b10e84ba0fab48293 WatchSource:0}: Error finding container 5a45e3b3e1b9c33e7f1145ad728f4870c9c3874e4640591b10e84ba0fab48293: Status 404 returned error can't find the container with id 5a45e3b3e1b9c33e7f1145ad728f4870c9c3874e4640591b10e84ba0fab48293
Oct 30 05:44:13.250416 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.250380    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:13.308259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.308227    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"]
Oct 30 05:44:13.311742 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.311718    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:13.328811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.328783    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-fffaa8\""
Oct 30 05:44:13.331155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.331121    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"]
Oct 30 05:44:13.381411 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.381345    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-fffaa8-dockercfg-bg2s8\""
Oct 30 05:44:13.403865 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.403838    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6l9r\" (UniqueName: \"kubernetes.io/projected/d929b643-076d-4366-8b66-6093f79d44ff-kube-api-access-j6l9r\") pod \"payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk\" (UID: \"d929b643-076d-4366-8b66-6093f79d44ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:13.404012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.403936    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d929b643-076d-4366-8b66-6093f79d44ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk\" (UID: \"d929b643-076d-4366-8b66-6093f79d44ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:13.426302 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.426275    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:13.434901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.434877    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"]
Oct 30 05:44:13.438579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.438564    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:13.459410 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.459383    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"]
Oct 30 05:44:13.505130 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.505101    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6l9r\" (UniqueName: \"kubernetes.io/projected/d929b643-076d-4366-8b66-6093f79d44ff-kube-api-access-j6l9r\") pod \"payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk\" (UID: \"d929b643-076d-4366-8b66-6093f79d44ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:13.505613 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.505230    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d929b643-076d-4366-8b66-6093f79d44ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk\" (UID: \"d929b643-076d-4366-8b66-6093f79d44ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:13.506043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.506020    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d929b643-076d-4366-8b66-6093f79d44ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk\" (UID: \"d929b643-076d-4366-8b66-6093f79d44ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:13.528396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.528374    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6l9r\" (UniqueName: \"kubernetes.io/projected/d929b643-076d-4366-8b66-6093f79d44ff-kube-api-access-j6l9r\") pod \"payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk\" (UID: \"d929b643-076d-4366-8b66-6093f79d44ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:13.588279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.588242    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:13.605952 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.605918    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1c6f7-eadf-4103-8708-8ec58c98146f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4\" (UID: \"61e1c6f7-eadf-4103-8708-8ec58c98146f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:13.606082 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.605954    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gqw\" (UniqueName: \"kubernetes.io/projected/61e1c6f7-eadf-4103-8708-8ec58c98146f-kube-api-access-p2gqw\") pod \"payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4\" (UID: \"61e1c6f7-eadf-4103-8708-8ec58c98146f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:13.625370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.625347    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:13.645940 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.645906    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4" event={"ID":"9f53076d-3434-4341-bd0c-836f77c610c7","Type":"ContainerStarted","Data":"4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c"}
Oct 30 05:44:13.646086 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.645999    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:13.647628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.647604    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8" event={"ID":"f40cfd76-bd78-423e-8ad7-ef3fee34eb52","Type":"ContainerStarted","Data":"6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2"}
Oct 30 05:44:13.647738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.647644    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8" event={"ID":"f40cfd76-bd78-423e-8ad7-ef3fee34eb52","Type":"ContainerStarted","Data":"5a45e3b3e1b9c33e7f1145ad728f4870c9c3874e4640591b10e84ba0fab48293"}
Oct 30 05:44:13.647738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.647694    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:13.698204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.698139    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4" podStartSLOduration=2.5807251559999997 podStartE2EDuration="4.698120933s" totalImagesPullingTime="2.117395777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:13.697980693 +0000 UTC m=+768.529017901" watchObservedRunningTime="2026-10-30 05:44:13.698120933 +0000 UTC m=+768.529158149"
Oct 30 05:44:13.707438 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.707412    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1c6f7-eadf-4103-8708-8ec58c98146f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4\" (UID: \"61e1c6f7-eadf-4103-8708-8ec58c98146f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:13.707555 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.707445    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gqw\" (UniqueName: \"kubernetes.io/projected/61e1c6f7-eadf-4103-8708-8ec58c98146f-kube-api-access-p2gqw\") pod \"payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4\" (UID: \"61e1c6f7-eadf-4103-8708-8ec58c98146f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:13.708106 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.708087    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1c6f7-eadf-4103-8708-8ec58c98146f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4\" (UID: \"61e1c6f7-eadf-4103-8708-8ec58c98146f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:13.733037 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.733007    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gqw\" (UniqueName: \"kubernetes.io/projected/61e1c6f7-eadf-4103-8708-8ec58c98146f-kube-api-access-p2gqw\") pod \"payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4\" (UID: \"61e1c6f7-eadf-4103-8708-8ec58c98146f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:13.745103 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.745053    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8" podStartSLOduration=1.7450389830000002 podStartE2EDuration="1.745038983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:13.743628697 +0000 UTC m=+768.574665917" watchObservedRunningTime="2026-10-30 05:44:13.745038983 +0000 UTC m=+768.576076201"
Oct 30 05:44:13.750815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.750794    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:13.843353 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.843311    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:13.938617 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:13.938595    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"]
Oct 30 05:44:13.940245 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:13.940211    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd929b643_076d_4366_8b66_6093f79d44ff.slice/crio-c3ca94ad6d4a27cc19111954dd1146fbcf21eaca55d935529af1f05098b5cc3c WatchSource:0}: Error finding container c3ca94ad6d4a27cc19111954dd1146fbcf21eaca55d935529af1f05098b5cc3c: Status 404 returned error can't find the container with id c3ca94ad6d4a27cc19111954dd1146fbcf21eaca55d935529af1f05098b5cc3c
Oct 30 05:44:14.059811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.059779    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"]
Oct 30 05:44:14.060689 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:14.060667    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e1c6f7_eadf_4103_8708_8ec58c98146f.slice/crio-ffdf53a87877dc799d4441aa9aa6785a8e38ee75200b349fe0a888c3807e4b53 WatchSource:0}: Error finding container ffdf53a87877dc799d4441aa9aa6785a8e38ee75200b349fe0a888c3807e4b53: Status 404 returned error can't find the container with id ffdf53a87877dc799d4441aa9aa6785a8e38ee75200b349fe0a888c3807e4b53
Oct 30 05:44:14.654381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.654348    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" event={"ID":"d929b643-076d-4366-8b66-6093f79d44ff","Type":"ContainerStarted","Data":"e1b69b5fb0f701318b105cd007732a71fa5d09a0e9526e5194eefc7a30534c04"}
Oct 30 05:44:14.654840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.654409    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" event={"ID":"d929b643-076d-4366-8b66-6093f79d44ff","Type":"ContainerStarted","Data":"c3ca94ad6d4a27cc19111954dd1146fbcf21eaca55d935529af1f05098b5cc3c"}
Oct 30 05:44:14.654840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.654584    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:14.656237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.656204    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4" event={"ID":"61e1c6f7-eadf-4103-8708-8ec58c98146f","Type":"ContainerStarted","Data":"b6b253f3ae5cb373149dafbb17186783158c89ceb380359dd69783b09c5f02e8"}
Oct 30 05:44:14.656502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.656487    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:14.656573 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.656518    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4" event={"ID":"61e1c6f7-eadf-4103-8708-8ec58c98146f","Type":"ContainerStarted","Data":"ffdf53a87877dc799d4441aa9aa6785a8e38ee75200b349fe0a888c3807e4b53"}
Oct 30 05:44:14.701920 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.701863    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" podStartSLOduration=1.7018454520000001 podStartE2EDuration="1.701845452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:14.697688099 +0000 UTC m=+769.528725318" watchObservedRunningTime="2026-10-30 05:44:14.701845452 +0000 UTC m=+769.532882672"
Oct 30 05:44:14.756462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.756409    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4" podStartSLOduration=1.756397124 podStartE2EDuration="1.756397124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:14.754998445 +0000 UTC m=+769.586035744" watchObservedRunningTime="2026-10-30 05:44:14.756397124 +0000 UTC m=+769.587434341"
Oct 30 05:44:14.916350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:14.916262    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:15.013974 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.013944    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"]
Oct 30 05:44:15.017636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.017619    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:15.040058 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.040037    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a14091-dockercfg-6ns5h\""
Oct 30 05:44:15.040777 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.040762    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a14091\""
Oct 30 05:44:15.054392 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.054366    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"]
Oct 30 05:44:15.123222 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.121096    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:15.123222 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.122688    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcdx6\" (UniqueName: \"kubernetes.io/projected/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-kube-api-access-qcdx6\") pod \"payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w\" (UID: \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:15.123222 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.122747    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w\" (UID: \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:15.223954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.223869    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcdx6\" (UniqueName: \"kubernetes.io/projected/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-kube-api-access-qcdx6\") pod \"payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w\" (UID: \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:15.223954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.223927    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w\" (UID: \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:15.224731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.224706    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w\" (UID: \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:15.247652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.247620    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcdx6\" (UniqueName: \"kubernetes.io/projected/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-kube-api-access-qcdx6\") pod \"payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w\" (UID: \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:15.330198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.330152    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:15.578122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.578043    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:15.629943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.629923    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"]
Oct 30 05:44:15.632005 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:15.631975    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2e1c1a_1396_4fe4_9a7e_f77f0b7fed02.slice/crio-3c2a4286912e6c8c4df72f040940f24eff885855fa0a7eb126fc6cc0b759dc70 WatchSource:0}: Error finding container 3c2a4286912e6c8c4df72f040940f24eff885855fa0a7eb126fc6cc0b759dc70: Status 404 returned error can't find the container with id 3c2a4286912e6c8c4df72f040940f24eff885855fa0a7eb126fc6cc0b759dc70
Oct 30 05:44:15.662823 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.662789    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w" event={"ID":"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02","Type":"ContainerStarted","Data":"3c2a4286912e6c8c4df72f040940f24eff885855fa0a7eb126fc6cc0b759dc70"}
Oct 30 05:44:15.745095 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.745067    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:15.956601 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:15.956562    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:16.670557 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:16.670519    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w" event={"ID":"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02","Type":"ContainerStarted","Data":"75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d"}
Oct 30 05:44:16.670945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:16.670602    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:16.713123 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:16.713072    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w" podStartSLOduration=2.713058949 podStartE2EDuration="2.713058949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:16.711370034 +0000 UTC m=+771.542407252" watchObservedRunningTime="2026-10-30 05:44:16.713058949 +0000 UTC m=+771.544096166"
Oct 30 05:44:17.671986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:17.671955    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:17.849195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:17.849141    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:18.006277 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:18.006166    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:19.736595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:19.736556    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:21.427306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.427273    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"]
Oct 30 05:44:21.431086 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.431071    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.461863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.461832    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a14091-serving-cert\""
Oct 30 05:44:21.462109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.462088    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 05:44:21.462226 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.462212    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7wbd8\""
Oct 30 05:44:21.464033 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.464007    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"]
Oct 30 05:44:21.474992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.474964    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwsz\" (UniqueName: \"kubernetes.io/projected/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-kube-api-access-2fwsz\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.475100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.475013    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.475168 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.475095    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.475168 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.475156    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-metrics-certs\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.576257 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.576221    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwsz\" (UniqueName: \"kubernetes.io/projected/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-kube-api-access-2fwsz\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.576418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.576278    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.576418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.576310    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.576418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.576346    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-metrics-certs\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.577106 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.577045    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.578947 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.578925    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.579097 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.578997    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-metrics-certs\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.588187 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.588157    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:21.604481 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.604457    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwsz\" (UniqueName: \"kubernetes.io/projected/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-kube-api-access-2fwsz\") pod \"maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.744483 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.744409    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:21.774660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:21.774630    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:22.056368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:22.056346    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"]
Oct 30 05:44:22.058307 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:22.058276    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f4efea_a882_4792_a1ca_2c4b5ebcd44d.slice/crio-30be399f4af84fdab693df1eefd4ecc7015143a4e0d2c31e93b949b5a6f8b9cf WatchSource:0}: Error finding container 30be399f4af84fdab693df1eefd4ecc7015143a4e0d2c31e93b949b5a6f8b9cf: Status 404 returned error can't find the container with id 30be399f4af84fdab693df1eefd4ecc7015143a4e0d2c31e93b949b5a6f8b9cf
Oct 30 05:44:22.703839 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:22.703799    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn" event={"ID":"57f4efea-a882-4792-a1ca-2c4b5ebcd44d","Type":"ContainerStarted","Data":"30be399f4af84fdab693df1eefd4ecc7015143a4e0d2c31e93b949b5a6f8b9cf"}
Oct 30 05:44:23.768196 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:23.768149    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:23.924871 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:23.924833    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:24.658695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:24.658667    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 05:44:24.658887 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:24.658744    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 05:44:24.718080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:24.718048    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn" event={"ID":"57f4efea-a882-4792-a1ca-2c4b5ebcd44d","Type":"ContainerStarted","Data":"abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c"}
Oct 30 05:44:24.718263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:24.718207    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:24.793994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:24.793941    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn" podStartSLOduration=1.57604465 podStartE2EDuration="3.793927199s" totalImagesPullingTime="2.217882549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:24.792336378 +0000 UTC m=+779.623373596" watchObservedRunningTime="2026-10-30 05:44:24.793927199 +0000 UTC m=+779.624964416"
Oct 30 05:44:25.220158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:25.220129    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:25.666295 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:25.666269    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 05:44:25.755930 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:25.755897    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:26.006993 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:26.006915    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:26.182006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:26.181949    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:26.674093 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:26.674062    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 05:44:27.531154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.531121    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"]
Oct 30 05:44:27.535150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.535118    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.580913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.580632    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"kube-root-ca.crt\""
Oct 30 05:44:27.580913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.580767    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"openshift-service-ca.crt\""
Oct 30 05:44:27.580913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.580881    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"default-dockercfg-bv9bw\""
Oct 30 05:44:27.581559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.581442    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"simulator-w4-698ab2-kserve-self-signed-certs\""
Oct 30 05:44:27.584927 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.584896    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"]
Oct 30 05:44:27.629731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.629696    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-model-cache\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.629854 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.629813    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tmp-dir\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.629908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.629857    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-home\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.629908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.629891    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-dshm\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.629979 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.629953    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tls-certs\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.630010 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.629986    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9glk9\" (UniqueName: \"kubernetes.io/projected/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-kube-api-access-9glk9\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.685206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.679763    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:27.687206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.687092    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 05:44:27.731242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731210    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-home\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.731429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731277    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-dshm\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.731429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731317    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tls-certs\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.731429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731352    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9glk9\" (UniqueName: \"kubernetes.io/projected/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-kube-api-access-9glk9\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.731594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731498    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-model-cache\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.731698 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731672    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tmp-dir\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.731830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731812    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-model-cache\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.731912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731674    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-home\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.732004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.731986    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tmp-dir\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.733871 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.733845    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-dshm\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.733975 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.733956    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tls-certs\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.789158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.789079    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9glk9\" (UniqueName: \"kubernetes.io/projected/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-kube-api-access-9glk9\") pod \"simulator-w4-698ab2-kserve-877d8b4b5-5w45w\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") " pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.847564 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.847532    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:27.886031 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:27.886002    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:28.183462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:28.183435    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"]
Oct 30 05:44:28.184484 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:28.184456    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f1f8bc_ea7d_49cd_8e2b_9ae9b9ced109.slice/crio-b3b4d75c687734a4732b42cbbb57cfca76a2a5b7c234a5490cfbc429b26efdab WatchSource:0}: Error finding container b3b4d75c687734a4732b42cbbb57cfca76a2a5b7c234a5490cfbc429b26efdab: Status 404 returned error can't find the container with id b3b4d75c687734a4732b42cbbb57cfca76a2a5b7c234a5490cfbc429b26efdab
Oct 30 05:44:28.742380 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:28.742338    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" event={"ID":"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109","Type":"ContainerStarted","Data":"b3b4d75c687734a4732b42cbbb57cfca76a2a5b7c234a5490cfbc429b26efdab"}
Oct 30 05:44:30.005289 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:30.005206    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:30.289082 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:30.288995    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:30.728142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:30.728115    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 05:44:30.754832 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:30.754800    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" event={"ID":"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109","Type":"ContainerStarted","Data":"8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9"}
Oct 30 05:44:30.815982 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:30.815925    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podStartSLOduration=2.299143655 podStartE2EDuration="3.815911218s" totalImagesPullingTime="1.516767563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:30.813967107 +0000 UTC m=+785.645004324" watchObservedRunningTime="2026-10-30 05:44:30.815911218 +0000 UTC m=+785.646948435"
Oct 30 05:44:30.848270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:30.848242    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:30.981697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:30.981610    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:31.179207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:31.175745    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:31.349570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:31.349488    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:32.180643 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:32.180562    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:32.392761 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:44:32.392729    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:44:34.744765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:34.744694    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:35.639343 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.639314    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:44:35.653977 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.639368    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:44:35.653977 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.647383    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:35.796142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.796106    5250 generic.go:356] "Generic (PLEG): container finished" podID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerID="0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e" exitCode=0
Oct 30 05:44:35.796548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.796199    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" event={"ID":"8ae41e05-46f8-418b-ab84-968a0e5a23b1","Type":"ContainerDied","Data":"0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e"}
Oct 30 05:44:35.796594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.796552    5250 scope.go:122] "RemoveContainer" containerID="0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e"
Oct 30 05:44:35.797838 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.797811    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:44:35.797931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.797881    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:44:35.797931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.797918    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:44:35.832960 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:35.832937    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:36.341401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:36.341355    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:36.516569 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:36.516533    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:36.804894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:36.804862    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" event={"ID":"8ae41e05-46f8-418b-ab84-968a0e5a23b1","Type":"ContainerStarted","Data":"0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9"}
Oct 30 05:44:37.048451 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:37.048421    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:37.241845 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:37.241766    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:37.242084 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:37.242056    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 05:44:37.242158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:37.242117    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 05:44:37.848353 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:37.848317    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:37.888265 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:37.888237    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:38.242166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:38.242125    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 05:44:38.242370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:38.242207    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 05:44:38.828232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:38.828196    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:44:39.065102 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:39.065066    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:39.241941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:39.241906    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 05:44:39.242117 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:39.241959    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 05:44:40.025775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.025741    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"]
Oct 30 05:44:40.029066 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.029048    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.084251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.073443    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"openshift-service-ca.crt\""
Oct 30 05:44:40.084251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.073586    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"simulator-w1-1d8c2f-kserve-self-signed-certs\""
Oct 30 05:44:40.084251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.073698    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"default-dockercfg-lkcsn\""
Oct 30 05:44:40.084251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.076350    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"kube-root-ca.crt\""
Oct 30 05:44:40.084251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.076947    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"]
Oct 30 05:44:40.116509 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.116477    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:40.199398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.199356    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bacfe6b-fd8d-485c-8f23-63d665188be5-tls-certs\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.199584 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.199425    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-model-cache\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.199584 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.199524    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-tmp-dir\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.199584 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.199569    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcbt\" (UniqueName: \"kubernetes.io/projected/4bacfe6b-fd8d-485c-8f23-63d665188be5-kube-api-access-ppcbt\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.199713 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.199668    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-dshm\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.199713 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.199707    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-home\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.242203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.242143    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 05:44:40.242364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.242221    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 05:44:40.301003 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.300915    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-dshm\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.301003 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.300956    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-home\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.301003 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.300989    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bacfe6b-fd8d-485c-8f23-63d665188be5-tls-certs\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.301350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.301022    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-model-cache\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.301350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.301048    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-tmp-dir\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.301350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.301075    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcbt\" (UniqueName: \"kubernetes.io/projected/4bacfe6b-fd8d-485c-8f23-63d665188be5-kube-api-access-ppcbt\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.301572 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.301465    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-home\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.301572 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.301489    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-model-cache\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.301572 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.301531    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-tmp-dir\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.303473 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.303450    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-dshm\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.303758 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.303742    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bacfe6b-fd8d-485c-8f23-63d665188be5-tls-certs\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.324017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.323985    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcbt\" (UniqueName: \"kubernetes.io/projected/4bacfe6b-fd8d-485c-8f23-63d665188be5-kube-api-access-ppcbt\") pod \"simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") " pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.342715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.342689    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:40.680471 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.678814    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"]
Oct 30 05:44:40.683493 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:40.683449    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bacfe6b_fd8d_485c_8f23_63d665188be5.slice/crio-7ef98b63518220d0b578336be6811b249adf8a89a07e3ab9f88eb472597f7d0c WatchSource:0}: Error finding container 7ef98b63518220d0b578336be6811b249adf8a89a07e3ab9f88eb472597f7d0c: Status 404 returned error can't find the container with id 7ef98b63518220d0b578336be6811b249adf8a89a07e3ab9f88eb472597f7d0c
Oct 30 05:44:40.831370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.831335    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" event={"ID":"4bacfe6b-fd8d-485c-8f23-63d665188be5","Type":"ContainerStarted","Data":"bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9"}
Oct 30 05:44:40.831370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.831376    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" event={"ID":"4bacfe6b-fd8d-485c-8f23-63d665188be5","Type":"ContainerStarted","Data":"7ef98b63518220d0b578336be6811b249adf8a89a07e3ab9f88eb472597f7d0c"}
Oct 30 05:44:40.894815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.894569    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podStartSLOduration=1.894551228 podStartE2EDuration="1.894551228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:40.89208392 +0000 UTC m=+795.723121138" watchObservedRunningTime="2026-10-30 05:44:40.894551228 +0000 UTC m=+795.725588446"
Oct 30 05:44:40.895265 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:40.895237    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:41.120442 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:41.120349    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:41.242302 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:41.242266    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 05:44:41.242456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:41.242331    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 05:44:41.343032 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:41.342997    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:41.519038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:41.519003    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:41.706347 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:41.706315    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:42.318628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:42.318568    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:42.390197 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:42.390154    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:42.390594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:42.390566    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:42.391592 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:42.391574    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 05:44:42.495430 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:42.495387    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:43.205226 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:43.205173    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:43.481200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:43.481093    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:45.345088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:45.345057    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:46.778653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:46.778615    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:47.268717 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:47.268675    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:47.495606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:47.495576    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:47.670092 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:47.670059    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:49.254303 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:49.254268    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:50.343898 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:50.343860    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:50.356710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:50.356686    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:50.884741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:50.884703    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:50.891336 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:50.891308    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:44:52.696696 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:52.696665    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:52.873748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:52.873714    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:53.649689 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.649651    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"]
Oct 30 05:44:53.659112 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.659085    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.673214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.673169    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"]
Oct 30 05:44:53.682355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.682332    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"default-dockercfg-nchgr\""
Oct 30 05:44:53.685062 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.685044    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"simulator-w5-be78f2-kserve-self-signed-certs\""
Oct 30 05:44:53.694425 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.694407    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"openshift-service-ca.crt\""
Oct 30 05:44:53.694507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.694454    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"kube-root-ca.crt\""
Oct 30 05:44:53.732118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.732077    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-tmp-dir\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.732611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.732173    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-dshm\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.732611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.732229    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-model-cache\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.732611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.732287    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-home\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.732611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.732385    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnww8\" (UniqueName: \"kubernetes.io/projected/39e5f085-5e01-47a9-a1a8-322c88830138-kube-api-access-bnww8\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.732611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.732414    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e5f085-5e01-47a9-a1a8-322c88830138-tls-certs\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.761096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.761062    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:53.833355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.833326    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnww8\" (UniqueName: \"kubernetes.io/projected/39e5f085-5e01-47a9-a1a8-322c88830138-kube-api-access-bnww8\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.833528 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.833361    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e5f085-5e01-47a9-a1a8-322c88830138-tls-certs\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.833528 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.833417    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-tmp-dir\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.833528 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.833458    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-dshm\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.833528 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.833482    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-model-cache\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.833742 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.833540    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-home\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.833974 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.833950    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-home\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.834145 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.834037    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-model-cache\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.834367 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.834298    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-tmp-dir\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.835906 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.835875    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-dshm\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.836598 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.836579    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e5f085-5e01-47a9-a1a8-322c88830138-tls-certs\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.856377 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.856352    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnww8\" (UniqueName: \"kubernetes.io/projected/39e5f085-5e01-47a9-a1a8-322c88830138-kube-api-access-bnww8\") pod \"simulator-w5-be78f2-kserve-7bdc44476b-82964\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") " pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:53.972990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:53.972910    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:54.290126 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:54.290091    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"]
Oct 30 05:44:54.291240 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:54.291208    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39e5f085_5e01_47a9_a1a8_322c88830138.slice/crio-671e508cd207bf29a04f5b2863fe43f6cb37b954ff38e5ca9d4dc8cf171e9b8f WatchSource:0}: Error finding container 671e508cd207bf29a04f5b2863fe43f6cb37b954ff38e5ca9d4dc8cf171e9b8f: Status 404 returned error can't find the container with id 671e508cd207bf29a04f5b2863fe43f6cb37b954ff38e5ca9d4dc8cf171e9b8f
Oct 30 05:44:54.604424 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:54.604313    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:54.905508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:54.905473    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" event={"ID":"39e5f085-5e01-47a9-a1a8-322c88830138","Type":"ContainerStarted","Data":"c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9"}
Oct 30 05:44:54.905937 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:54.905518    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" event={"ID":"39e5f085-5e01-47a9-a1a8-322c88830138","Type":"ContainerStarted","Data":"671e508cd207bf29a04f5b2863fe43f6cb37b954ff38e5ca9d4dc8cf171e9b8f"}
Oct 30 05:44:54.958251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:54.958201    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podStartSLOduration=1.958188003 podStartE2EDuration="1.958188003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:54.956120311 +0000 UTC m=+809.787157528" watchObservedRunningTime="2026-10-30 05:44:54.958188003 +0000 UTC m=+809.789225211"
Oct 30 05:44:54.973046 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:54.973022    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:44:56.315720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:56.315682    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:57.620531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.620496    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"]
Oct 30 05:44:57.625014 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.624989    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.659200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.659148    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"openshift-service-ca.crt\""
Oct 30 05:44:57.659484 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.659255    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"kube-root-ca.crt\""
Oct 30 05:44:57.659484 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.659338    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"default-dockercfg-279cs\""
Oct 30 05:44:57.660952 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.660928    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"simulator-w3-fffaa8-kserve-self-signed-certs\""
Oct 30 05:44:57.661370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.661350    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"]
Oct 30 05:44:57.671113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.671086    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qddtb\" (UniqueName: \"kubernetes.io/projected/1bc6c877-c2ee-4620-a543-e740ed43437e-kube-api-access-qddtb\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.671235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.671121    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-tmp-dir\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.671282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.671231    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc6c877-c2ee-4620-a543-e740ed43437e-tls-certs\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.671282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.671273    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-home\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.671353 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.671299    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-dshm\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.671353 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.671322    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-model-cache\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.738368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.738330    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:57.772082 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772049    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-dshm\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.772276 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772092    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-model-cache\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.772276 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772214    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qddtb\" (UniqueName: \"kubernetes.io/projected/1bc6c877-c2ee-4620-a543-e740ed43437e-kube-api-access-qddtb\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.772276 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772256    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-tmp-dir\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.772462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772299    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc6c877-c2ee-4620-a543-e740ed43437e-tls-certs\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.772462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772405    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-home\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.772987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772624    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-model-cache\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.773114 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772678    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-tmp-dir\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.773114 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.772896    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-home\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.774534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.774510    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-dshm\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.775015 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.774999    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc6c877-c2ee-4620-a543-e740ed43437e-tls-certs\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.795742 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.795718    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qddtb\" (UniqueName: \"kubernetes.io/projected/1bc6c877-c2ee-4620-a543-e740ed43437e-kube-api-access-qddtb\") pod \"simulator-w3-fffaa8-kserve-bf469b6-9rgn7\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") " pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:57.938709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:57.938679    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:58.112110 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:58.112056    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:58.244149 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:58.244123    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"]
Oct 30 05:44:58.245582 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:44:58.245554    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc6c877_c2ee_4620_a543_e740ed43437e.slice/crio-9ffdbfb5c46a5b60eb0db07eaac374acc72d3dc9d382e7823090fd1c6b40a2a5 WatchSource:0}: Error finding container 9ffdbfb5c46a5b60eb0db07eaac374acc72d3dc9d382e7823090fd1c6b40a2a5: Status 404 returned error can't find the container with id 9ffdbfb5c46a5b60eb0db07eaac374acc72d3dc9d382e7823090fd1c6b40a2a5
Oct 30 05:44:58.929231 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:58.929173    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" event={"ID":"1bc6c877-c2ee-4620-a543-e740ed43437e","Type":"ContainerStarted","Data":"5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736"}
Oct 30 05:44:58.929652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:58.929240    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" event={"ID":"1bc6c877-c2ee-4620-a543-e740ed43437e","Type":"ContainerStarted","Data":"9ffdbfb5c46a5b60eb0db07eaac374acc72d3dc9d382e7823090fd1c6b40a2a5"}
Oct 30 05:44:58.939233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:58.939207    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:44:58.982379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:58.982335    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podStartSLOduration=1.982321775 podStartE2EDuration="1.982321775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:58.980551695 +0000 UTC m=+813.811588906" watchObservedRunningTime="2026-10-30 05:44:58.982321775 +0000 UTC m=+813.813358992"
Oct 30 05:44:59.913137 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:44:59.913096    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:00.206063 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.205972    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"]
Oct 30 05:45:00.209704 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.209682    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"]
Oct 30 05:45:00.209830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.209815    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:00.213089 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.213053    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:00.229908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.229885    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"]
Oct 30 05:45:00.239217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.238598    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 05:45:00.242955 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.242933    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"]
Oct 30 05:45:00.246853 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.246836    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:00.282123 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.282097    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"]
Oct 30 05:45:00.282251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.282148    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"]
Oct 30 05:45:00.297101 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.297072    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6mqn\" (UniqueName: \"kubernetes.io/projected/f69b1a6f-fa74-486b-b620-a612c68a8846-kube-api-access-w6mqn\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv\" (UID: \"f69b1a6f-fa74-486b-b620-a612c68a8846\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:00.297312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.297138    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf\" (UID: \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:00.297312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.297236    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfwg\" (UniqueName: \"kubernetes.io/projected/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-kube-api-access-xtfwg\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf\" (UID: \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:00.297312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.297276    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2rp\" (UniqueName: \"kubernetes.io/projected/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-kube-api-access-5p2rp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x\" (UID: \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:00.297312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.297302    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x\" (UID: \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:00.297493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.297425    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69b1a6f-fa74-486b-b620-a612c68a8846-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv\" (UID: \"f69b1a6f-fa74-486b-b620-a612c68a8846\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:00.398307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.398271    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfwg\" (UniqueName: \"kubernetes.io/projected/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-kube-api-access-xtfwg\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf\" (UID: \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:00.398461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.398316    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2rp\" (UniqueName: \"kubernetes.io/projected/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-kube-api-access-5p2rp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x\" (UID: \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:00.398461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.398335    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x\" (UID: \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:00.398461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.398395    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69b1a6f-fa74-486b-b620-a612c68a8846-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv\" (UID: \"f69b1a6f-fa74-486b-b620-a612c68a8846\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:00.398461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.398454    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6mqn\" (UniqueName: \"kubernetes.io/projected/f69b1a6f-fa74-486b-b620-a612c68a8846-kube-api-access-w6mqn\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv\" (UID: \"f69b1a6f-fa74-486b-b620-a612c68a8846\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:00.398700 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.398542    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf\" (UID: \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:00.399199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.399158    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf\" (UID: \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:00.399308 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.399161    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69b1a6f-fa74-486b-b620-a612c68a8846-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv\" (UID: \"f69b1a6f-fa74-486b-b620-a612c68a8846\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:00.399308 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.399216    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x\" (UID: \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:00.428818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.428793    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2rp\" (UniqueName: \"kubernetes.io/projected/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-kube-api-access-5p2rp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x\" (UID: \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:00.428953 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.428938    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfwg\" (UniqueName: \"kubernetes.io/projected/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-kube-api-access-xtfwg\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf\" (UID: \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:00.429077 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.429057    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6mqn\" (UniqueName: \"kubernetes.io/projected/f69b1a6f-fa74-486b-b620-a612c68a8846-kube-api-access-w6mqn\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv\" (UID: \"f69b1a6f-fa74-486b-b620-a612c68a8846\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:00.524717 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.524645    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:00.533693 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.533669    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:00.559754 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.559724    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:00.841035 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.840950    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"]
Oct 30 05:45:00.856221 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:45:00.856195    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720a6b23_2b24_4be5_9c71_ad6d0d8fd422.slice/crio-59276ee8fafd986a08f4fbfd64c6c17f1b29ad007d7822b83f56782fa0838066 WatchSource:0}: Error finding container 59276ee8fafd986a08f4fbfd64c6c17f1b29ad007d7822b83f56782fa0838066: Status 404 returned error can't find the container with id 59276ee8fafd986a08f4fbfd64c6c17f1b29ad007d7822b83f56782fa0838066
Oct 30 05:45:00.885273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.885247    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"]
Oct 30 05:45:00.888654 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:45:00.888627    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69b1a6f_fa74_486b_b620_a612c68a8846.slice/crio-ecbc5a53a0681da200c02bc6a02386985b60513a23b566d5202282a5ebd30ef7 WatchSource:0}: Error finding container ecbc5a53a0681da200c02bc6a02386985b60513a23b566d5202282a5ebd30ef7: Status 404 returned error can't find the container with id ecbc5a53a0681da200c02bc6a02386985b60513a23b566d5202282a5ebd30ef7
Oct 30 05:45:00.911504 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.911485    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"]
Oct 30 05:45:00.913462 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:45:00.913438    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bbb90f_ed64_4cc8_9455_4bbe427de1a6.slice/crio-1eeeb0385bed9e2b918aa0b493be422650856026f6aed540507e758e48463706 WatchSource:0}: Error finding container 1eeeb0385bed9e2b918aa0b493be422650856026f6aed540507e758e48463706: Status 404 returned error can't find the container with id 1eeeb0385bed9e2b918aa0b493be422650856026f6aed540507e758e48463706
Oct 30 05:45:00.940450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.940424    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" event={"ID":"f69b1a6f-fa74-486b-b620-a612c68a8846","Type":"ContainerStarted","Data":"ecbc5a53a0681da200c02bc6a02386985b60513a23b566d5202282a5ebd30ef7"}
Oct 30 05:45:00.941731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.941694    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" event={"ID":"54bbb90f-ed64-4cc8-9455-4bbe427de1a6","Type":"ContainerStarted","Data":"1eeeb0385bed9e2b918aa0b493be422650856026f6aed540507e758e48463706"}
Oct 30 05:45:00.943143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:00.943114    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" event={"ID":"720a6b23-2b24-4be5-9c71-ad6d0d8fd422","Type":"ContainerStarted","Data":"59276ee8fafd986a08f4fbfd64c6c17f1b29ad007d7822b83f56782fa0838066"}
Oct 30 05:45:01.942160 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:01.942085    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:02.380550 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:02.380439    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:45:03.970767 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:03.970677    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" event={"ID":"f69b1a6f-fa74-486b-b620-a612c68a8846","Type":"ContainerStarted","Data":"4f8c62970e80f920a83e20fff75c994b403cce077a8c05dacf5f53bfc6f68f34"}
Oct 30 05:45:03.973610 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:03.973582    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:45:03.976749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:03.976714    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" event={"ID":"54bbb90f-ed64-4cc8-9455-4bbe427de1a6","Type":"ContainerStarted","Data":"677bfaef6bd297b872ba3354c72b66509a720b351baea06ed254f9ee145ce1df"}
Oct 30 05:45:03.978659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:03.978629    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" event={"ID":"720a6b23-2b24-4be5-9c71-ad6d0d8fd422","Type":"ContainerStarted","Data":"e8f4ccb2a351c696363257f97f020689e19aaa5764c706eba0d9f536ea8ae831"}
Oct 30 05:45:03.989827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:03.989806    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:03.993515 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:03.993493    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:45:04.011615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.011574    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" podStartSLOduration=1.459846998 podStartE2EDuration="4.011562679s" totalImagesPullingTime="2.551715681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:04.011305612 +0000 UTC m=+818.842342830" watchObservedRunningTime="2026-10-30 05:45:04.011562679 +0000 UTC m=+818.842599893"
Oct 30 05:45:04.057167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.057111    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" podStartSLOduration=1.520656157 podStartE2EDuration="4.057093559s" totalImagesPullingTime="2.536437402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:04.055626865 +0000 UTC m=+818.886664085" watchObservedRunningTime="2026-10-30 05:45:04.057093559 +0000 UTC m=+818.888130777"
Oct 30 05:45:04.099224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.099158    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" podStartSLOduration=1.6482836939999999 podStartE2EDuration="4.099146105s" totalImagesPullingTime="2.450862411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:04.097481694 +0000 UTC m=+818.928518908" watchObservedRunningTime="2026-10-30 05:45:04.099146105 +0000 UTC m=+818.930183321"
Oct 30 05:45:04.970250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.970207    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:04.986682 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.986649    5250 generic.go:356] "Generic (PLEG): container finished" podID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerID="677bfaef6bd297b872ba3354c72b66509a720b351baea06ed254f9ee145ce1df" exitCode=60
Oct 30 05:45:04.987122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.986734    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" event={"ID":"54bbb90f-ed64-4cc8-9455-4bbe427de1a6","Type":"ContainerDied","Data":"677bfaef6bd297b872ba3354c72b66509a720b351baea06ed254f9ee145ce1df"}
Oct 30 05:45:04.987122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.987005    5250 scope.go:122] "RemoveContainer" containerID="677bfaef6bd297b872ba3354c72b66509a720b351baea06ed254f9ee145ce1df"
Oct 30 05:45:04.989044 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.989020    5250 generic.go:356] "Generic (PLEG): container finished" podID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerID="e8f4ccb2a351c696363257f97f020689e19aaa5764c706eba0d9f536ea8ae831" exitCode=60
Oct 30 05:45:04.989193 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.989165    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" event={"ID":"720a6b23-2b24-4be5-9c71-ad6d0d8fd422","Type":"ContainerDied","Data":"e8f4ccb2a351c696363257f97f020689e19aaa5764c706eba0d9f536ea8ae831"}
Oct 30 05:45:04.989496 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.989480    5250 scope.go:122] "RemoveContainer" containerID="e8f4ccb2a351c696363257f97f020689e19aaa5764c706eba0d9f536ea8ae831"
Oct 30 05:45:04.991738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.991716    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:45:04.992476 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.992449    5250 generic.go:356] "Generic (PLEG): container finished" podID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerID="4f8c62970e80f920a83e20fff75c994b403cce077a8c05dacf5f53bfc6f68f34" exitCode=60
Oct 30 05:45:04.992596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.992535    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" event={"ID":"f69b1a6f-fa74-486b-b620-a612c68a8846","Type":"ContainerDied","Data":"4f8c62970e80f920a83e20fff75c994b403cce077a8c05dacf5f53bfc6f68f34"}
Oct 30 05:45:04.992703 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:04.992681    5250 scope.go:122] "RemoveContainer" containerID="4f8c62970e80f920a83e20fff75c994b403cce077a8c05dacf5f53bfc6f68f34"
Oct 30 05:45:05.988133 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:05.988100    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:05.999685 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:05.999657    5250 generic.go:356] "Generic (PLEG): container finished" podID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerID="8e4083324bdc51937ab38bd3bed4257c32d1654f1ce3c18d6990b9bb3c142758" exitCode=60
Oct 30 05:45:05.999815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:05.999763    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" event={"ID":"54bbb90f-ed64-4cc8-9455-4bbe427de1a6","Type":"ContainerDied","Data":"8e4083324bdc51937ab38bd3bed4257c32d1654f1ce3c18d6990b9bb3c142758"}
Oct 30 05:45:05.999872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:05.999820    5250 scope.go:122] "RemoveContainer" containerID="677bfaef6bd297b872ba3354c72b66509a720b351baea06ed254f9ee145ce1df"
Oct 30 05:45:06.000075 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.000053    5250 scope.go:122] "RemoveContainer" containerID="8e4083324bdc51937ab38bd3bed4257c32d1654f1ce3c18d6990b9bb3c142758"
Oct 30 05:45:06.000376 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:06.000346    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf_odh-ai-gateway-infra(54bbb90f-ed64-4cc8-9455-4bbe427de1a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6"
Oct 30 05:45:06.002015 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.001992    5250 generic.go:356] "Generic (PLEG): container finished" podID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerID="21b38d869a81b257e02621d20486f66e6d82c091857716e4723901f208d08961" exitCode=60
Oct 30 05:45:06.002107 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.002069    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" event={"ID":"720a6b23-2b24-4be5-9c71-ad6d0d8fd422","Type":"ContainerDied","Data":"21b38d869a81b257e02621d20486f66e6d82c091857716e4723901f208d08961"}
Oct 30 05:45:06.002365 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.002346    5250 scope.go:122] "RemoveContainer" containerID="21b38d869a81b257e02621d20486f66e6d82c091857716e4723901f208d08961"
Oct 30 05:45:06.002623 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:06.002593    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x_odh-ai-gateway-infra(720a6b23-2b24-4be5-9c71-ad6d0d8fd422)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422"
Oct 30 05:45:06.004109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.004087    5250 generic.go:356] "Generic (PLEG): container finished" podID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerID="9f4771d25aea94bc4f80b6a268b83136f8a4ba64a8faa1fd9a0c72ebd3d71ada" exitCode=60
Oct 30 05:45:06.004243 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.004162    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" event={"ID":"f69b1a6f-fa74-486b-b620-a612c68a8846","Type":"ContainerDied","Data":"9f4771d25aea94bc4f80b6a268b83136f8a4ba64a8faa1fd9a0c72ebd3d71ada"}
Oct 30 05:45:06.004467 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.004450    5250 scope.go:122] "RemoveContainer" containerID="9f4771d25aea94bc4f80b6a268b83136f8a4ba64a8faa1fd9a0c72ebd3d71ada"
Oct 30 05:45:06.004683 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:06.004666    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv_odh-ai-gateway-infra(f69b1a6f-fa74-486b-b620-a612c68a8846)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846"
Oct 30 05:45:06.122441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.122420    5250 scope.go:122] "RemoveContainer" containerID="e8f4ccb2a351c696363257f97f020689e19aaa5764c706eba0d9f536ea8ae831"
Oct 30 05:45:06.140334 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.140309    5250 scope.go:122] "RemoveContainer" containerID="4f8c62970e80f920a83e20fff75c994b403cce077a8c05dacf5f53bfc6f68f34"
Oct 30 05:45:06.793492 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:06.793453    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:07.011532 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:07.011503    5250 scope.go:122] "RemoveContainer" containerID="21b38d869a81b257e02621d20486f66e6d82c091857716e4723901f208d08961"
Oct 30 05:45:07.011996 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:07.011765    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x_odh-ai-gateway-infra(720a6b23-2b24-4be5-9c71-ad6d0d8fd422)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422"
Oct 30 05:45:07.013229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:07.013206    5250 scope.go:122] "RemoveContainer" containerID="9f4771d25aea94bc4f80b6a268b83136f8a4ba64a8faa1fd9a0c72ebd3d71ada"
Oct 30 05:45:07.013436 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:07.013412    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv_odh-ai-gateway-infra(f69b1a6f-fa74-486b-b620-a612c68a8846)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846"
Oct 30 05:45:07.014747 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:07.014732    5250 scope.go:122] "RemoveContainer" containerID="8e4083324bdc51937ab38bd3bed4257c32d1654f1ce3c18d6990b9bb3c142758"
Oct 30 05:45:07.014956 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:07.014940    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf_odh-ai-gateway-infra(54bbb90f-ed64-4cc8-9455-4bbe427de1a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6"
Oct 30 05:45:07.939791 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:07.939761    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:45:07.952429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:07.952402    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:45:08.031120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:08.031093    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:45:08.128790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:08.128757    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:09.893017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:09.892982    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:12.300674 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:12.300632    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:14.170465 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:14.170427    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:15.227449 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.227414    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"]
Oct 30 05:45:15.231963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.231941    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.264299 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.264261    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"premium-w1-1d8c2f-kserve-self-signed-certs\""
Oct 30 05:45:15.267725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.267695    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"]
Oct 30 05:45:15.348871 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.348841    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-dshm\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.349031 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.348879    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-tmp-dir\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.349073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.349035    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-model-cache\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.349108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.349098    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-home\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.349150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.349135    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff40c6-828b-4662-b068-5938a065ef90-tls-certs\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.349279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.349259    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vw56\" (UniqueName: \"kubernetes.io/projected/2cff40c6-828b-4662-b068-5938a065ef90-kube-api-access-9vw56\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.392390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.392359    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:15.451006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.450966    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-dshm\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.451244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.451029    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-tmp-dir\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.451244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.451088    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-model-cache\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.451244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.451129    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-home\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.451244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.451202    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff40c6-828b-4662-b068-5938a065ef90-tls-certs\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.451484 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.451245    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vw56\" (UniqueName: \"kubernetes.io/projected/2cff40c6-828b-4662-b068-5938a065ef90-kube-api-access-9vw56\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.451804 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.451777    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-model-cache\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.451929 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.451906    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-home\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.452038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.452021    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-tmp-dir\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.454979 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.454954    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff40c6-828b-4662-b068-5938a065ef90-tls-certs\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.455523 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.455501    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-dshm\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.477542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.477457    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vw56\" (UniqueName: \"kubernetes.io/projected/2cff40c6-828b-4662-b068-5938a065ef90-kube-api-access-9vw56\") pod \"premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") " pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.544673 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.544642    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:15.857775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:15.857745    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"]
Oct 30 05:45:15.859255 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:45:15.859225    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cff40c6_828b_4662_b068_5938a065ef90.slice/crio-dca1dabe8eb9cdc7ef74da559d57c69837026d50d0d81897f2ff02e325c66e4b WatchSource:0}: Error finding container dca1dabe8eb9cdc7ef74da559d57c69837026d50d0d81897f2ff02e325c66e4b: Status 404 returned error can't find the container with id dca1dabe8eb9cdc7ef74da559d57c69837026d50d0d81897f2ff02e325c66e4b
Oct 30 05:45:16.066613 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:16.066573    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" event={"ID":"2cff40c6-828b-4662-b068-5938a065ef90","Type":"ContainerStarted","Data":"1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009"}
Oct 30 05:45:16.066613 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:16.066616    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" event={"ID":"2cff40c6-828b-4662-b068-5938a065ef90","Type":"ContainerStarted","Data":"dca1dabe8eb9cdc7ef74da559d57c69837026d50d0d81897f2ff02e325c66e4b"}
Oct 30 05:45:16.070465 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:16.070441    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:16.114073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:16.113986    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podStartSLOduration=1.11397219 podStartE2EDuration="1.11397219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:16.111705106 +0000 UTC m=+830.942742322" watchObservedRunningTime="2026-10-30 05:45:16.11397219 +0000 UTC m=+830.945009406"
Oct 30 05:45:16.545440 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:16.545408    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:17.274150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:17.274123    5250 scope.go:122] "RemoveContainer" containerID="9f4771d25aea94bc4f80b6a268b83136f8a4ba64a8faa1fd9a0c72ebd3d71ada"
Oct 30 05:45:17.931446 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:17.931409    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:18.078963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:18.078933    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" event={"ID":"f69b1a6f-fa74-486b-b620-a612c68a8846","Type":"ContainerStarted","Data":"4544ffcccbaea4a12755c04795e1b045ebf8063b9517f4bbf9f6e391e2a76b6e"}
Oct 30 05:45:18.274300 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:18.274220    5250 scope.go:122] "RemoveContainer" containerID="8e4083324bdc51937ab38bd3bed4257c32d1654f1ce3c18d6990b9bb3c142758"
Oct 30 05:45:18.274490 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:18.274299    5250 scope.go:122] "RemoveContainer" containerID="21b38d869a81b257e02621d20486f66e6d82c091857716e4723901f208d08961"
Oct 30 05:45:19.086550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:19.086512    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" event={"ID":"54bbb90f-ed64-4cc8-9455-4bbe427de1a6","Type":"ContainerStarted","Data":"16432971aba42f971006a1b79461b6886bc1fb81179f70070a2024982dd2f3b5"}
Oct 30 05:45:19.088661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:19.088636    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" event={"ID":"720a6b23-2b24-4be5-9c71-ad6d0d8fd422","Type":"ContainerStarted","Data":"e75369b9a6978c37f6c47831fbcec68575710423a0fcdb2f15c31d202400f402"}
Oct 30 05:45:19.090709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:19.090686    5250 generic.go:356] "Generic (PLEG): container finished" podID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerID="4544ffcccbaea4a12755c04795e1b045ebf8063b9517f4bbf9f6e391e2a76b6e" exitCode=60
Oct 30 05:45:19.090786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:19.090746    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" event={"ID":"f69b1a6f-fa74-486b-b620-a612c68a8846","Type":"ContainerDied","Data":"4544ffcccbaea4a12755c04795e1b045ebf8063b9517f4bbf9f6e391e2a76b6e"}
Oct 30 05:45:19.090841 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:19.090792    5250 scope.go:122] "RemoveContainer" containerID="9f4771d25aea94bc4f80b6a268b83136f8a4ba64a8faa1fd9a0c72ebd3d71ada"
Oct 30 05:45:19.091078 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:19.091062    5250 scope.go:122] "RemoveContainer" containerID="4544ffcccbaea4a12755c04795e1b045ebf8063b9517f4bbf9f6e391e2a76b6e"
Oct 30 05:45:19.091332 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:19.091304    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv_odh-ai-gateway-infra(f69b1a6f-fa74-486b-b620-a612c68a8846)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846"
Oct 30 05:45:19.165020 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:19.164993    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"]
Oct 30 05:45:19.902798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:19.902764    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:20.098850 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.098815    5250 generic.go:356] "Generic (PLEG): container finished" podID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerID="16432971aba42f971006a1b79461b6886bc1fb81179f70070a2024982dd2f3b5" exitCode=60
Oct 30 05:45:20.099362 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.098888    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" event={"ID":"54bbb90f-ed64-4cc8-9455-4bbe427de1a6","Type":"ContainerDied","Data":"16432971aba42f971006a1b79461b6886bc1fb81179f70070a2024982dd2f3b5"}
Oct 30 05:45:20.099362 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.098991    5250 scope.go:122] "RemoveContainer" containerID="8e4083324bdc51937ab38bd3bed4257c32d1654f1ce3c18d6990b9bb3c142758"
Oct 30 05:45:20.099949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.099926    5250 scope.go:122] "RemoveContainer" containerID="16432971aba42f971006a1b79461b6886bc1fb81179f70070a2024982dd2f3b5"
Oct 30 05:45:20.100477 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:20.100267    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf_odh-ai-gateway-infra(54bbb90f-ed64-4cc8-9455-4bbe427de1a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6"
Oct 30 05:45:20.102232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.102211    5250 generic.go:356] "Generic (PLEG): container finished" podID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerID="e75369b9a6978c37f6c47831fbcec68575710423a0fcdb2f15c31d202400f402" exitCode=60
Oct 30 05:45:20.102344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.102278    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" event={"ID":"720a6b23-2b24-4be5-9c71-ad6d0d8fd422","Type":"ContainerDied","Data":"e75369b9a6978c37f6c47831fbcec68575710423a0fcdb2f15c31d202400f402"}
Oct 30 05:45:20.102638 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.102621    5250 scope.go:122] "RemoveContainer" containerID="e75369b9a6978c37f6c47831fbcec68575710423a0fcdb2f15c31d202400f402"
Oct 30 05:45:20.102873 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:20.102856    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x_odh-ai-gateway-infra(720a6b23-2b24-4be5-9c71-ad6d0d8fd422)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422"
Oct 30 05:45:20.112793 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.112772    5250 scope.go:122] "RemoveContainer" containerID="21b38d869a81b257e02621d20486f66e6d82c091857716e4723901f208d08961"
Oct 30 05:45:20.161115 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.161039    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"]
Oct 30 05:45:20.223733 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.223702    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"]
Oct 30 05:45:20.259391 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.259369    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:20.398337 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.398306    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69b1a6f-fa74-486b-b620-a612c68a8846-kube-api-access-w6mqn\" (UniqueName: \"kubernetes.io/projected/f69b1a6f-fa74-486b-b620-a612c68a8846-kube-api-access-w6mqn\") pod \"f69b1a6f-fa74-486b-b620-a612c68a8846\" (UID: \"f69b1a6f-fa74-486b-b620-a612c68a8846\") "
Oct 30 05:45:20.398533 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.398408    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69b1a6f-fa74-486b-b620-a612c68a8846-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69b1a6f-fa74-486b-b620-a612c68a8846-maas-api-service-ca\") pod \"f69b1a6f-fa74-486b-b620-a612c68a8846\" (UID: \"f69b1a6f-fa74-486b-b620-a612c68a8846\") "
Oct 30 05:45:20.398817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.398791    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69b1a6f-fa74-486b-b620-a612c68a8846-maas-api-service-ca" pod "f69b1a6f-fa74-486b-b620-a612c68a8846" (UID: "f69b1a6f-fa74-486b-b620-a612c68a8846"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:45:20.400833 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.400809    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69b1a6f-fa74-486b-b620-a612c68a8846-kube-api-access-w6mqn" pod "f69b1a6f-fa74-486b-b620-a612c68a8846" (UID: "f69b1a6f-fa74-486b-b620-a612c68a8846"). InnerVolumeSpecName "kube-api-access-w6mqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:45:20.499378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.499299    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6mqn\" (UniqueName: \"kubernetes.io/projected/f69b1a6f-fa74-486b-b620-a612c68a8846-kube-api-access-w6mqn\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:45:20.499378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.499326    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69b1a6f-fa74-486b-b620-a612c68a8846-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:45:20.965080 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:20.965044    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:21.114670 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.114637    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv" event={"ID":"f69b1a6f-fa74-486b-b620-a612c68a8846","Type":"ContainerDied","Data":"ecbc5a53a0681da200c02bc6a02386985b60513a23b566d5202282a5ebd30ef7"}
Oct 30 05:45:21.115096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.114707    5250 scope.go:122] "RemoveContainer" containerID="4544ffcccbaea4a12755c04795e1b045ebf8063b9517f4bbf9f6e391e2a76b6e"
Oct 30 05:45:21.115096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.114654    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"
Oct 30 05:45:21.189410 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.189380    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"]
Oct 30 05:45:21.203221 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.202048    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29888985-6kxqv"]
Oct 30 05:45:21.280833 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.280801    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" path="/var/lib/kubelet/pods/f69b1a6f-fa74-486b-b620-a612c68a8846/volumes"
Oct 30 05:45:21.293558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.293530    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:21.296397 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.296377    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:21.412022 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.411992    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-kube-api-access-5p2rp\" (UniqueName: \"kubernetes.io/projected/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-kube-api-access-5p2rp\") pod \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\" (UID: \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\") "
Oct 30 05:45:21.412200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.412034    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-maas-api-service-ca\") pod \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\" (UID: \"720a6b23-2b24-4be5-9c71-ad6d0d8fd422\") "
Oct 30 05:45:21.412200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.412140    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-kube-api-access-xtfwg\" (UniqueName: \"kubernetes.io/projected/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-kube-api-access-xtfwg\") pod \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\" (UID: \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\") "
Oct 30 05:45:21.412200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.412194    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-maas-api-service-ca\") pod \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\" (UID: \"54bbb90f-ed64-4cc8-9455-4bbe427de1a6\") "
Oct 30 05:45:21.412471 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.412438    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-maas-api-service-ca" pod "720a6b23-2b24-4be5-9c71-ad6d0d8fd422" (UID: "720a6b23-2b24-4be5-9c71-ad6d0d8fd422"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:45:21.412626 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.412604    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:45:21.412773 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.412740    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-maas-api-service-ca" pod "54bbb90f-ed64-4cc8-9455-4bbe427de1a6" (UID: "54bbb90f-ed64-4cc8-9455-4bbe427de1a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:45:21.414533 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.414509    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-kube-api-access-xtfwg" pod "54bbb90f-ed64-4cc8-9455-4bbe427de1a6" (UID: "54bbb90f-ed64-4cc8-9455-4bbe427de1a6"). InnerVolumeSpecName "kube-api-access-xtfwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:45:21.414660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.414648    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-kube-api-access-5p2rp" pod "720a6b23-2b24-4be5-9c71-ad6d0d8fd422" (UID: "720a6b23-2b24-4be5-9c71-ad6d0d8fd422"). InnerVolumeSpecName "kube-api-access-5p2rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:45:21.514136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.514055    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2rp\" (UniqueName: \"kubernetes.io/projected/720a6b23-2b24-4be5-9c71-ad6d0d8fd422-kube-api-access-5p2rp\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:45:21.514136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.514095    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtfwg\" (UniqueName: \"kubernetes.io/projected/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-kube-api-access-xtfwg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:45:21.514136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.514112    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bbb90f-ed64-4cc8-9455-4bbe427de1a6-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:45:21.768642 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:21.768546    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:22.122872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.122793    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"
Oct 30 05:45:22.123332 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.122798    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf" event={"ID":"54bbb90f-ed64-4cc8-9455-4bbe427de1a6","Type":"ContainerDied","Data":"1eeeb0385bed9e2b918aa0b493be422650856026f6aed540507e758e48463706"}
Oct 30 05:45:22.123332 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.122935    5250 scope.go:122] "RemoveContainer" containerID="16432971aba42f971006a1b79461b6886bc1fb81179f70070a2024982dd2f3b5"
Oct 30 05:45:22.124250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.124233    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"
Oct 30 05:45:22.124349 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.124229    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x" event={"ID":"720a6b23-2b24-4be5-9c71-ad6d0d8fd422","Type":"ContainerDied","Data":"59276ee8fafd986a08f4fbfd64c6c17f1b29ad007d7822b83f56782fa0838066"}
Oct 30 05:45:22.133207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.133167    5250 scope.go:122] "RemoveContainer" containerID="e75369b9a6978c37f6c47831fbcec68575710423a0fcdb2f15c31d202400f402"
Oct 30 05:45:22.177244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.177217    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"]
Oct 30 05:45:22.188519 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.188498    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29888985-nn4jf"]
Oct 30 05:45:22.221387 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.221365    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"]
Oct 30 05:45:22.228112 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:22.228088    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29888985-vpm9x"]
Oct 30 05:45:23.279716 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:23.279674    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" path="/var/lib/kubelet/pods/54bbb90f-ed64-4cc8-9455-4bbe427de1a6/volumes"
Oct 30 05:45:23.280261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:23.280242    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" path="/var/lib/kubelet/pods/720a6b23-2b24-4be5-9c71-ad6d0d8fd422/volumes"
Oct 30 05:45:23.631782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:23.631665    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:25.288250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:25.288214    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:25.544886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:25.544810    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:25.560883 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:25.560857    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:26.157805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:26.157775    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:45:27.021997 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:27.021950    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:28.893589 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:28.893534    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:29.022267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022227    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"]
Oct 30 05:45:29.022793 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022774    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:29.022793 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022794    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022815    5250 cpu_manager.go:414] "containerMap: removing container" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022821    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022832    5250 cpu_manager.go:414] "containerMap: removing container" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022840    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022852    5250 cpu_manager.go:414] "containerMap: removing container" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022856    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022875    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022880    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022891    5250 cpu_manager.go:414] "containerMap: removing container" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:29.022948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022895    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:29.023450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022966    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:29.023450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022978    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:29.023450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022988    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:29.023450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.022998    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:29.023450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.023006    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:29.023450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.023018    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:29.023450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.023027    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:29.023450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.023037    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:29.031972 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.031949    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.044717 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.044689    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"]
Oct 30 05:45:29.046835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.046657    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"openshift-service-ca.crt\""
Oct 30 05:45:29.046835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.046707    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"kube-root-ca.crt\""
Oct 30 05:45:29.046835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.046757    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"premium-w6-64123f-kserve-self-signed-certs\""
Oct 30 05:45:29.054123 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.054100    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"default-dockercfg-d9ktx\""
Oct 30 05:45:29.088619 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.088596    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57dc45f-74b8-4009-a354-2c8d32d570ce-tls-certs\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.088731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.088636    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-dshm\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.088773 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.088732    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-home\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.088773 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.088764    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-model-cache\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.088869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.088795    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-tmp-dir\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.088869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.088858    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx48s\" (UniqueName: \"kubernetes.io/projected/c57dc45f-74b8-4009-a354-2c8d32d570ce-kube-api-access-dx48s\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.121640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.121615    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:29.189668 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.189574    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-home\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.189668 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.189620    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-model-cache\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.189668 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.189656    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-tmp-dir\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.190076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.189686    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx48s\" (UniqueName: \"kubernetes.io/projected/c57dc45f-74b8-4009-a354-2c8d32d570ce-kube-api-access-dx48s\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.190076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.189727    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57dc45f-74b8-4009-a354-2c8d32d570ce-tls-certs\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.190076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.189769    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-dshm\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.190076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.190023    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-home\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.190349 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.190093    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-model-cache\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.190349 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.190152    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-tmp-dir\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.192514 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.192481    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-dshm\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.192657 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.192640    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57dc45f-74b8-4009-a354-2c8d32d570ce-tls-certs\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.214873 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.214778    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx48s\" (UniqueName: \"kubernetes.io/projected/c57dc45f-74b8-4009-a354-2c8d32d570ce-kube-api-access-dx48s\") pod \"premium-w6-64123f-kserve-866798d69b-pckkw\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") " pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.349603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.349571    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:29.651099 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:29.651071    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"]
Oct 30 05:45:29.653084 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:45:29.653052    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc57dc45f_74b8_4009_a354_2c8d32d570ce.slice/crio-2d29368968e7386ae10e126a9590840928581091b36dc9a41727dfee9fbcde9f WatchSource:0}: Error finding container 2d29368968e7386ae10e126a9590840928581091b36dc9a41727dfee9fbcde9f: Status 404 returned error can't find the container with id 2d29368968e7386ae10e126a9590840928581091b36dc9a41727dfee9fbcde9f
Oct 30 05:45:30.169892 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:30.169852    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" event={"ID":"c57dc45f-74b8-4009-a354-2c8d32d570ce","Type":"ContainerStarted","Data":"44daaf2ae28f59dd469cf887717554a3756c598f75cf97f82786394779cc1f40"}
Oct 30 05:45:30.170351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:30.169899    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" event={"ID":"c57dc45f-74b8-4009-a354-2c8d32d570ce","Type":"ContainerStarted","Data":"2d29368968e7386ae10e126a9590840928581091b36dc9a41727dfee9fbcde9f"}
Oct 30 05:45:30.213964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:30.213915    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podStartSLOduration=2.213900473 podStartE2EDuration="2.213900473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:30.212994922 +0000 UTC m=+845.044032138" watchObservedRunningTime="2026-10-30 05:45:30.213900473 +0000 UTC m=+845.044937690"
Oct 30 05:45:30.349700 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:30.349670    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:30.599666 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:30.599566    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:31.489118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:31.489086    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:32.373670 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:45:32.373635    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:45:32.379719 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:32.379689    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:34.015559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:34.015526    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:35.767368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:35.767332    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:37.359116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:37.359086    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:38.830870 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.830835    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"]
Oct 30 05:45:38.831506 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.831487    5250 cpu_manager.go:414] "containerMap: removing container" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:38.831506 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.831507    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720a6b23-2b24-4be5-9c71-ad6d0d8fd422" containerName="cleanup"
Oct 30 05:45:38.831667 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.831520    5250 cpu_manager.go:414] "containerMap: removing container" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:38.831667 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.831525    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bbb90f-ed64-4cc8-9455-4bbe427de1a6" containerName="cleanup"
Oct 30 05:45:38.831667 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.831550    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:38.831667 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.831555    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:38.831667 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.831616    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69b1a6f-fa74-486b-b620-a612c68a8846" containerName="cleanup"
Oct 30 05:45:38.834854 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.834836    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.861099 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.861073    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"premium-w5-be78f2-kserve-self-signed-certs\""
Oct 30 05:45:38.863371 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.863346    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"]
Oct 30 05:45:38.879520 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.879488    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-tmp-dir\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.879626 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.879543    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hh6f\" (UniqueName: \"kubernetes.io/projected/04314bc8-7661-4875-b931-94dcfdef9ee6-kube-api-access-8hh6f\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.879671 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.879620    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-home\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.879722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.879673    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-dshm\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.879760 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.879722    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-model-cache\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.879835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.879817    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04314bc8-7661-4875-b931-94dcfdef9ee6-tls-certs\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.924236 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.924203    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:38.980705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.980668    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-model-cache\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.980901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.980750    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04314bc8-7661-4875-b931-94dcfdef9ee6-tls-certs\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.980901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.980789    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-tmp-dir\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.980901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.980833    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hh6f\" (UniqueName: \"kubernetes.io/projected/04314bc8-7661-4875-b931-94dcfdef9ee6-kube-api-access-8hh6f\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.980901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.980864    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-home\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.981125 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.980930    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-dshm\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.981125 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.981093    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-model-cache\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.981392 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.981362    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-home\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.981525 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.981420    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-tmp-dir\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.983846 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.983813    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-dshm\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:38.984044 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:38.984024    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04314bc8-7661-4875-b931-94dcfdef9ee6-tls-certs\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:39.004271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:39.004219    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hh6f\" (UniqueName: \"kubernetes.io/projected/04314bc8-7661-4875-b931-94dcfdef9ee6-kube-api-access-8hh6f\") pod \"premium-w5-be78f2-kserve-599bc94ff9-898tb\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") " pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:39.070440 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:39.070370    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:39.147638 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:39.147610    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:39.349872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:39.349843    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:39.362489 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:39.362468    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:39.452415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:39.452369    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"]
Oct 30 05:45:39.454721 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:45:39.454701    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04314bc8_7661_4875_b931_94dcfdef9ee6.slice/crio-cd16f28f8d307c9ede22134108498f8990f8d23b5c32232636662c847de985a3 WatchSource:0}: Error finding container cd16f28f8d307c9ede22134108498f8990f8d23b5c32232636662c847de985a3: Status 404 returned error can't find the container with id cd16f28f8d307c9ede22134108498f8990f8d23b5c32232636662c847de985a3
Oct 30 05:45:40.224502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:40.224460    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" event={"ID":"04314bc8-7661-4875-b931-94dcfdef9ee6","Type":"ContainerStarted","Data":"4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb"}
Oct 30 05:45:40.224914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:40.224514    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" event={"ID":"04314bc8-7661-4875-b931-94dcfdef9ee6","Type":"ContainerStarted","Data":"cd16f28f8d307c9ede22134108498f8990f8d23b5c32232636662c847de985a3"}
Oct 30 05:45:40.236268 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:40.236245    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:45:40.266190 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:40.266135    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podStartSLOduration=2.266122653 podStartE2EDuration="2.266122653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:40.265135765 +0000 UTC m=+855.096172985" watchObservedRunningTime="2026-10-30 05:45:40.266122653 +0000 UTC m=+855.097159888"
Oct 30 05:45:40.627069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:40.626973    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:41.148173 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:41.148139    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:42.318422 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:42.318384    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:43.887390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:43.887355    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:44.046173 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:44.046142    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:45.696702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:45.696663    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:47.511604 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:47.511558    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:49.058291 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:49.058259    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:49.147951 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:49.147900    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:49.166153 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:49.166129    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:49.284804 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:49.284778    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:45:50.750385 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:50.750339    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:52.328134 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.328099    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"]
Oct 30 05:45:52.331283 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.331263    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.347490 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.347461    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"]
Oct 30 05:45:52.348738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.348716    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"kube-root-ca.crt\""
Oct 30 05:45:52.356526 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.356506    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"openshift-service-ca.crt\""
Oct 30 05:45:52.356815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.356798    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:52.387770 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.387747    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"premium-w0-a14091-kserve-self-signed-certs\""
Oct 30 05:45:52.392269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.392248    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"default-dockercfg-c4xwk\""
Oct 30 05:45:52.411642 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.411611    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-dshm\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.411743 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.411647    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tmp-dir\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.411809 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.411754    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tls-certs\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.411809 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.411782    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfsl8\" (UniqueName: \"kubernetes.io/projected/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-kube-api-access-kfsl8\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.411934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.411921    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-home\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.411969 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.411954    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-model-cache\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.505898 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.505859    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:52.513313 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.513279    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-model-cache\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.513604 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.513583    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-dshm\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.513725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.513712    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tmp-dir\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.513886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.513873    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tls-certs\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.514278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.514249    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfsl8\" (UniqueName: \"kubernetes.io/projected/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-kube-api-access-kfsl8\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.514549 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.514534    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-home\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.515106 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.515085    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-home\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.515278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.515172    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tmp-dir\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.515501 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.515476    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-model-cache\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.519266 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.519244    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tls-certs\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.522677 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.522651    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-dshm\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.541559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.541533    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfsl8\" (UniqueName: \"kubernetes.io/projected/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-kube-api-access-kfsl8\") pod \"premium-w0-a14091-kserve-78b6ffb4f4-mfjsh\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") " pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.642975 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.642939    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:52.950562 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:52.950537    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"]
Oct 30 05:45:52.952972 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:45:52.952934    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c0f2cb3_53e8_4162_aefe_3b99db4ba678.slice/crio-1d7fea01b5f4505d0086a805235366054b9fb48ecf5cd9fd57b7d28dc29db4c7 WatchSource:0}: Error finding container 1d7fea01b5f4505d0086a805235366054b9fb48ecf5cd9fd57b7d28dc29db4c7: Status 404 returned error can't find the container with id 1d7fea01b5f4505d0086a805235366054b9fb48ecf5cd9fd57b7d28dc29db4c7
Oct 30 05:45:53.294606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:53.294522    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" event={"ID":"9c0f2cb3-53e8-4162-aefe-3b99db4ba678","Type":"ContainerStarted","Data":"46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1"}
Oct 30 05:45:53.294606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:53.294565    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" event={"ID":"9c0f2cb3-53e8-4162-aefe-3b99db4ba678","Type":"ContainerStarted","Data":"1d7fea01b5f4505d0086a805235366054b9fb48ecf5cd9fd57b7d28dc29db4c7"}
Oct 30 05:45:53.382967 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:53.382915    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podStartSLOduration=1.3829021 podStartE2EDuration="1.3829021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:53.379750488 +0000 UTC m=+868.210787708" watchObservedRunningTime="2026-10-30 05:45:53.3829021 +0000 UTC m=+868.213939316"
Oct 30 05:45:53.643474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:53.643447    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:45:54.463411 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:54.463337    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:56.195728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:56.195694    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:56.522524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:56.522438    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:57.865351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:57.865303    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:59.964498 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:45:59.964462    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:01.485363 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:01.485248    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:02.373266 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:46:02.373232    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:46:02.643924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:02.643893    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:46:02.656919 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:02.656897    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:46:03.324973 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:03.324939    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:03.361476 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:03.361439    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:46:04.977728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:04.977695    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:06.534708 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:06.534666    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:08.348830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:08.348788    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:09.912470 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:09.912427    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:11.642192 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:11.642137    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:12.581598 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:12.581563    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:13.331131 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:13.331098    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:15.404355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:15.404320    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:16.479288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.479257    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"]
Oct 30 05:46:16.483101 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.483081    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.497597 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.497575    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1d8c2f-openshift-default-dockercfg-7kqf6\""
Oct 30 05:46:16.511300 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.511269    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"]
Oct 30 05:46:16.551043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551011    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.551243 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551049    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.551243 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551070    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.551379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551290    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2fs\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-kube-api-access-fm2fs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.551379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551329    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.551463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551396    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.551463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551437    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.551463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551457    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.551637 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.551478    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.652652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.652617    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2fs\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-kube-api-access-fm2fs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.652829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.652658    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.652829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.652684    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.652829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.652724    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.652829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.652755    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.652829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.652786    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.653091 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.652881    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.653091 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.652925    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.653223 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.653095    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.653223 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.653130    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.653367 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.653337    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.653427 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.653373    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.653531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.653508    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.653590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.653535    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.655319 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.655289    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.655627 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.655610    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.676012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.675991    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.679872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.679850    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2fs\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-kube-api-access-fm2fs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.797685 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.797605    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:16.884633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:16.884582    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:17.121035 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:17.121010    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"]
Oct 30 05:46:17.122954 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:46:17.122928    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83b2a65_5c23_4a43_aa87_ba0806f7b7a8.slice/crio-8c654d9e7acaa85e11653d804370b907be26727b09cd15abe7dce416706213d3 WatchSource:0}: Error finding container 8c654d9e7acaa85e11653d804370b907be26727b09cd15abe7dce416706213d3: Status 404 returned error can't find the container with id 8c654d9e7acaa85e11653d804370b907be26727b09cd15abe7dce416706213d3
Oct 30 05:46:17.125334 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:17.125305    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:17.125405 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:17.125389    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:17.125446 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:17.125424    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:17.429314 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:17.429270    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" event={"ID":"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8","Type":"ContainerStarted","Data":"f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928"}
Oct 30 05:46:17.429517 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:17.429336    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" event={"ID":"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8","Type":"ContainerStarted","Data":"8c654d9e7acaa85e11653d804370b907be26727b09cd15abe7dce416706213d3"}
Oct 30 05:46:17.485229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:17.485157    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" podStartSLOduration=1.485142446 podStartE2EDuration="1.485142446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:17.483534661 +0000 UTC m=+892.314571877" watchObservedRunningTime="2026-10-30 05:46:17.485142446 +0000 UTC m=+892.316179659"
Oct 30 05:46:17.798302 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:17.798213    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:18.800199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:18.800151    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 05:46:18.800617 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:18.800216    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 05:46:18.910934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:18.910901    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:19.798961 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:19.798916    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 05:46:19.799156 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:19.798979    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 05:46:20.798614 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:20.798577    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 05:46:20.798996 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:20.798642    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 05:46:20.889358 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:20.889298    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:21.798910 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:21.798871    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 05:46:21.799351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:21.798937    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 05:46:22.587215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:22.587163    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:22.798317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:22.798281    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 05:46:22.798500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:22.798379    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 05:46:23.803098 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:23.803063    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:23.803550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:23.803425    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:23.804035 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:23.804018    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 05:46:24.484398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:24.484354    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:25.374887 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:25.374855    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:46:25.376699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:25.376675    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:46:25.968552 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:25.968521    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:27.671700 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:27.671657    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:29.092822 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:29.092787    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:30.981409 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:30.981340    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:31.355800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.355729    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"]
Oct 30 05:46:31.360491 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.360474    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.375855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.375825    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-64123f-openshift-default-dockercfg-msqjh\""
Oct 30 05:46:31.389295 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.389268    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"]
Oct 30 05:46:31.439136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439106    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.439136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439139    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.439345 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439200    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.439345 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439310    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ede062-4c03-4bf5-b803-01ce16478c33-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.439413 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439358    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzg56\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-kube-api-access-dzg56\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.439447 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439412    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.439491 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439451    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.439525 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439503    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ede062-4c03-4bf5-b803-01ce16478c33-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.439558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.439551    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.540894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.540857    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.540894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.540901    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.540945    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.540977    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ede062-4c03-4bf5-b803-01ce16478c33-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541013    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzg56\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-kube-api-access-dzg56\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541045    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541068    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541147    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ede062-4c03-4bf5-b803-01ce16478c33-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541218    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541380    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541445    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541470    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.541701 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.541559    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.542331 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.542307    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ede062-4c03-4bf5-b803-01ce16478c33-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.543480 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.543459    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.543698 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.543676    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ede062-4c03-4bf5-b803-01ce16478c33-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.565096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.565062    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.565768 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.565749    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzg56\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-kube-api-access-dzg56\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.585444 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.585421    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:31.674519 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.674482    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:31.994021 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.993999    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"]
Oct 30 05:46:31.995687 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:46:31.995665    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ede062_4c03_4bf5_b803_01ce16478c33.slice/crio-8e10f1e92f8bc5db7216abd610704b7ff4a509db3975aea4068e21da8ca14e60 WatchSource:0}: Error finding container 8e10f1e92f8bc5db7216abd610704b7ff4a509db3975aea4068e21da8ca14e60: Status 404 returned error can't find the container with id 8e10f1e92f8bc5db7216abd610704b7ff4a509db3975aea4068e21da8ca14e60
Oct 30 05:46:31.997914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.997884    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:31.997990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.997942    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:31.997990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:31.997972    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:32.376256 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:46:32.376138    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:46:32.518671 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:32.518638    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:32.532540 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:32.532507    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" event={"ID":"91ede062-4c03-4bf5-b803-01ce16478c33","Type":"ContainerStarted","Data":"80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8"}
Oct 30 05:46:32.532733 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:32.532578    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" event={"ID":"91ede062-4c03-4bf5-b803-01ce16478c33","Type":"ContainerStarted","Data":"8e10f1e92f8bc5db7216abd610704b7ff4a509db3975aea4068e21da8ca14e60"}
Oct 30 05:46:32.577790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:32.577742    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" podStartSLOduration=1.5777285490000001 podStartE2EDuration="1.577728549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:32.576031436 +0000 UTC m=+907.407068653" watchObservedRunningTime="2026-10-30 05:46:32.577728549 +0000 UTC m=+907.408765766"
Oct 30 05:46:32.675420 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:32.675373    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:33.678205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:33.677297    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 05:46:33.678205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:33.677376    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 05:46:33.995663 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:33.995569    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:34.680225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:34.678721    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 05:46:34.680225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:34.678777    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 05:46:35.681025 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:35.680983    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 05:46:35.684487 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:35.684441    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 05:46:35.876068 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:35.875860    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:36.678595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:36.678553    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 05:46:36.678899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:36.678867    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 05:46:37.399456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:37.399406    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:37.680395 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:37.680284    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 05:46:37.680395 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:37.680365    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 05:46:38.902480 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:38.902433    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:38.902979 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:38.902761    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:38.904324 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:38.904303    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 05:46:39.064337 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:39.064309    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:40.373370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:40.373336    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:40.570231 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:40.570161    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:42.167885 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:42.167851    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:44.963640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:44.963606    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:46.547534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:46.547501    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:47.183623 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.183589    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"]
Oct 30 05:46:47.187431 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.187415    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.204341 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.204320    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-be78f2-openshift-default-dockercfg-8twx9\""
Oct 30 05:46:47.223869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.223840    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"]
Oct 30 05:46:47.231084 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231053    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51df8972-896d-4d45-9975-6cf732c8f699-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.231284 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231101    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdw5k\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-kube-api-access-hdw5k\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.231284 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231169    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.231284 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231264    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.231469 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231288    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.231469 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231304    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.231469 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231391    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.231469 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231440    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51df8972-896d-4d45-9975-6cf732c8f699-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.231680 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.231472    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333051 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333009    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333077    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333092    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333114    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333527 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333501    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333792 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333759    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333787    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333814    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333869    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51df8972-896d-4d45-9975-6cf732c8f699-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333996 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333902    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.333996 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333947    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51df8972-896d-4d45-9975-6cf732c8f699-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.334082 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.333991    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdw5k\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-kube-api-access-hdw5k\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.334254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.334231    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.334575 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.334558    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51df8972-896d-4d45-9975-6cf732c8f699-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.335912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.335894    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.336490 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.336470    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51df8972-896d-4d45-9975-6cf732c8f699-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.356909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.356880    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdw5k\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-kube-api-access-hdw5k\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.357843 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.357822    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.500823 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.500740    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:47.819448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.819425    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"]
Oct 30 05:46:47.821643 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:46:47.821615    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51df8972_896d_4d45_9975_6cf732c8f699.slice/crio-17fadbdf25b61b929c796676d2067b0c7594495d82daa9dbd0d3e9abf500621b WatchSource:0}: Error finding container 17fadbdf25b61b929c796676d2067b0c7594495d82daa9dbd0d3e9abf500621b: Status 404 returned error can't find the container with id 17fadbdf25b61b929c796676d2067b0c7594495d82daa9dbd0d3e9abf500621b
Oct 30 05:46:47.824059 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.824029    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:47.824143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.824096    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:47.824143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:47.824121    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:46:48.392710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:48.392668    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:48.652990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:48.652911    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" event={"ID":"51df8972-896d-4d45-9975-6cf732c8f699","Type":"ContainerStarted","Data":"3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01"}
Oct 30 05:46:48.652990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:48.652964    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" event={"ID":"51df8972-896d-4d45-9975-6cf732c8f699","Type":"ContainerStarted","Data":"17fadbdf25b61b929c796676d2067b0c7594495d82daa9dbd0d3e9abf500621b"}
Oct 30 05:46:48.718754 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:48.718700    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" podStartSLOduration=1.71868536 podStartE2EDuration="1.71868536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:48.718483016 +0000 UTC m=+923.549520246" watchObservedRunningTime="2026-10-30 05:46:48.71868536 +0000 UTC m=+923.549722577"
Oct 30 05:46:49.500994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:49.500951    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:49.502699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:49.502658    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 05:46:49.502899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:49.502860    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 05:46:49.837348 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:49.837250    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:50.501301 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:50.501269    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 05:46:50.501712 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:50.501320    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 05:46:51.485876 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:51.485834    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:51.501312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:51.501281    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 05:46:51.501712 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:51.501344    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 05:46:52.501549 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:52.501513    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 05:46:52.501948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:52.501567    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 05:46:52.843597 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:52.843510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:53.501653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:53.501619    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 05:46:53.502077 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:53.501695    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 05:46:54.344155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:54.344122    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:54.505050 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:54.505019    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:54.505470 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:54.505436    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:54.506217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:54.506170    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 05:46:55.952286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:55.952253    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:58.274410 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:46:58.274378    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:00.477575 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.477543    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"]
Oct 30 05:47:00.481738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.481715    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.485193 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.485154    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:00.510280 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.510253    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"distinct-w1-1d8c2f-kserve-self-signed-certs\""
Oct 30 05:47:00.513014 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.512987    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"]
Oct 30 05:47:00.602520 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.602492    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tls-certs\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.602520 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.602524    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-dshm\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.602752 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.602568    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-model-cache\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.602752 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.602721    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-home\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.602752 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.602746    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tmp-dir\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.602901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.602879    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbns\" (UniqueName: \"kubernetes.io/projected/3cab2e63-c729-4d45-9fd7-7748cb5add6b-kube-api-access-sgbns\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.628418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.628390    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:00.704422 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.704387    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbns\" (UniqueName: \"kubernetes.io/projected/3cab2e63-c729-4d45-9fd7-7748cb5add6b-kube-api-access-sgbns\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.704599 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.704470    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tls-certs\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.704599 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.704495    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-dshm\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.704599 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.704547    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-model-cache\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.704788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.704621    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-home\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.704788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.704646    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tmp-dir\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.705142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.705113    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-home\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.705288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.705140    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tmp-dir\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.705398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.705381    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-model-cache\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.707357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.707329    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-dshm\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.707600 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.707584    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tls-certs\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.728158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.728097    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbns\" (UniqueName: \"kubernetes.io/projected/3cab2e63-c729-4d45-9fd7-7748cb5add6b-kube-api-access-sgbns\") pod \"distinct-w1-1d8c2f-kserve-f98459d48-2gwrm\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:00.795759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:00.795728    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:01.097513 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.097486    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"]
Oct 30 05:47:01.098374 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:47:01.098349    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cab2e63_c729_4d45_9fd7_7748cb5add6b.slice/crio-b1d7bd94428ac73b8cf20b715870c1c3d4db11ec6c088144214589307987b6ca WatchSource:0}: Error finding container b1d7bd94428ac73b8cf20b715870c1c3d4db11ec6c088144214589307987b6ca: Status 404 returned error can't find the container with id b1d7bd94428ac73b8cf20b715870c1c3d4db11ec6c088144214589307987b6ca
Oct 30 05:47:01.645594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.645556    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"]
Oct 30 05:47:01.649467 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.649446    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.673150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.673122    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"]
Oct 30 05:47:01.748160 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.748132    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"]
Oct 30 05:47:01.752043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.752022    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.761394 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.761369    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" event={"ID":"3cab2e63-c729-4d45-9fd7-7748cb5add6b","Type":"ContainerStarted","Data":"c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6"}
Oct 30 05:47:01.761480 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.761402    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" event={"ID":"3cab2e63-c729-4d45-9fd7-7748cb5add6b","Type":"ContainerStarted","Data":"b1d7bd94428ac73b8cf20b715870c1c3d4db11ec6c088144214589307987b6ca"}
Oct 30 05:47:01.766382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.766363    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-698ab2-openshift-default-dockercfg-fl985\""
Oct 30 05:47:01.776745 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.776718    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"]
Oct 30 05:47:01.796018 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.795995    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:01.815950 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.815922    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.816079 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.815978    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.816079 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.816054    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.816271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.816126    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.816271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.816161    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.816361 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.816310    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52e50f-e4ab-482b-925a-2dfba4994c19-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.816361 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.816351    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.816467 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.816367    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkz7v\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-kube-api-access-jkz7v\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.816467 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.816418    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.834822 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.834780    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podStartSLOduration=1.834767653 podStartE2EDuration="1.834767653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:01.833086447 +0000 UTC m=+936.664123665" watchObservedRunningTime="2026-10-30 05:47:01.834767653 +0000 UTC m=+936.665804866"
Oct 30 05:47:01.917500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917425    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.917500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917477    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52e50f-e4ab-482b-925a-2dfba4994c19-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.917500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917498    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.917720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917649    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.917720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917697    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.917720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917714    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkz7v\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-kube-api-access-jkz7v\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.917870 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917737    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.917870 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917753    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkl7z\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-kube-api-access-xkl7z\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.917870 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917864    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918021 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917881    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.918021 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917933    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918021 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917960    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918021 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.917989    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.918273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918053    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.918273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918077    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918101    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.918273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918140    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918164    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:01.918273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918225    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52e50f-e4ab-482b-925a-2dfba4994c19-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918247    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918306    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918338    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.918818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.918484    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.920443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.920415    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.920553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.920481    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.940764 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.940742    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.940957 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.940939    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkz7v\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-kube-api-access-jkz7v\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:01.962929 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:01.962897    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:02.019168 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019115    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkl7z\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-kube-api-access-xkl7z\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.019429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019210    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.019429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019254    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.019429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019300    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.019429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019326    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.019429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019352    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.019429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019405    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.019429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019432    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.023815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.019450    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.023815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.020694    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.023815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.020701    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.023815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.020858    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.023815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.021047    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.023815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.021460    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.023815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.023039    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.024980 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.024960    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.042428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.042397    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.042568 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.042518    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkl7z\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-kube-api-access-xkl7z\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.066781 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.066752    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:02.164831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.159843    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:02.287360 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.287335    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"]
Oct 30 05:47:02.289240 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:47:02.289208    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b52e50f_e4ab_482b_925a_2dfba4994c19.slice/crio-0f793bf52db8bbc82f388662f74a6b00dce0a1106848f5f8b1ca83dfcac1c104 WatchSource:0}: Error finding container 0f793bf52db8bbc82f388662f74a6b00dce0a1106848f5f8b1ca83dfcac1c104: Status 404 returned error can't find the container with id 0f793bf52db8bbc82f388662f74a6b00dce0a1106848f5f8b1ca83dfcac1c104
Oct 30 05:47:02.291785 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.291755    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:02.291876 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.291817    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:02.291876 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.291847    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:02.376573 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:47:02.376453    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:47:02.379421 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.379395    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"]
Oct 30 05:47:02.383588 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.383568    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.389637 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.389610    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"]
Oct 30 05:47:02.392024 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:47:02.391999    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0596f462_3f2b_4f5b_8c22_10ccd7ad5a8b.slice/crio-e79f78737f26ebe6d049d93fb0a2be01f4e61bf70dd8134b64dc72c2a3d89752 WatchSource:0}: Error finding container e79f78737f26ebe6d049d93fb0a2be01f4e61bf70dd8134b64dc72c2a3d89752: Status 404 returned error can't find the container with id e79f78737f26ebe6d049d93fb0a2be01f4e61bf70dd8134b64dc72c2a3d89752
Oct 30 05:47:02.395043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.395011    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:02.395146 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.395070    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:02.395146 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.395098    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:02.397995 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.397968    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e7a506-openshift-default-dockercfg-552kl\""
Oct 30 05:47:02.406364 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.405408    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"]
Oct 30 05:47:02.440262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440230    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.440429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440282    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.440565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440449    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.440565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440497    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.440565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440560    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61de2173-e6ea-4c2c-b3f5-c48200fef197-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.440861 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440625    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.440861 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440656    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrxm\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-kube-api-access-4lrxm\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.440861 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440687    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.440861 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.440777    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542349    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542441    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61de2173-e6ea-4c2c-b3f5-c48200fef197-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542513    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542547    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrxm\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-kube-api-access-4lrxm\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542575    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542608    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542694    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542728    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.542825    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.543776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.543285    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.544756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.544242    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.544756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.544500    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.544756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.544582    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61de2173-e6ea-4c2c-b3f5-c48200fef197-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.544756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.544722    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.546708 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.546569    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.546987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.546967    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.567289 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.567256    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrxm\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-kube-api-access-4lrxm\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.567731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.567711    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.700574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.700545    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:02.812201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.806211    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" event={"ID":"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b","Type":"ContainerStarted","Data":"e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915"}
Oct 30 05:47:02.812201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.806288    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" event={"ID":"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b","Type":"ContainerStarted","Data":"e79f78737f26ebe6d049d93fb0a2be01f4e61bf70dd8134b64dc72c2a3d89752"}
Oct 30 05:47:02.812201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.810347    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" event={"ID":"6b52e50f-e4ab-482b-925a-2dfba4994c19","Type":"ContainerStarted","Data":"d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9"}
Oct 30 05:47:02.812201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.810392    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" event={"ID":"6b52e50f-e4ab-482b-925a-2dfba4994c19","Type":"ContainerStarted","Data":"0f793bf52db8bbc82f388662f74a6b00dce0a1106848f5f8b1ca83dfcac1c104"}
Oct 30 05:47:02.887359 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.857166    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podStartSLOduration=1.85714806 podStartE2EDuration="1.85714806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:02.853664086 +0000 UTC m=+937.684701304" watchObservedRunningTime="2026-10-30 05:47:02.85714806 +0000 UTC m=+937.688185278"
Oct 30 05:47:02.902241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.902112    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podStartSLOduration=1.902091319 podStartE2EDuration="1.902091319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:02.900734948 +0000 UTC m=+937.731772165" watchObservedRunningTime="2026-10-30 05:47:02.902091319 +0000 UTC m=+937.733128534"
Oct 30 05:47:02.982213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:02.976579    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:03.041344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.041316    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"]
Oct 30 05:47:03.042688 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:47:03.042663    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61de2173_e6ea_4c2c_b3f5_c48200fef197.slice/crio-ef764ec7142d53f129a293c44b323dfc8d40c9a15ee67296e53e0c27b2e397ee WatchSource:0}: Error finding container ef764ec7142d53f129a293c44b323dfc8d40c9a15ee67296e53e0c27b2e397ee: Status 404 returned error can't find the container with id ef764ec7142d53f129a293c44b323dfc8d40c9a15ee67296e53e0c27b2e397ee
Oct 30 05:47:03.045876 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.045844    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:03.045992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.045922    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:03.045992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.045963    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:03.072661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.072531    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:03.076679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.076285    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 05:47:03.076679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.076356    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 05:47:03.718621 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.718572    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"]
Oct 30 05:47:03.722808 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.722777    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.748819 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.747301    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"]
Oct 30 05:47:03.765203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.764822    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.765203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.764925    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.765203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.764957    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.765203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.764979    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.765203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.765015    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dh6\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-kube-api-access-z8dh6\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.765203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.765043    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.765203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.765138    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.765733 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.765224    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.765733 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.765287    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.820693 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.820653    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:03.830198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.829772    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" event={"ID":"61de2173-e6ea-4c2c-b3f5-c48200fef197","Type":"ContainerStarted","Data":"2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2"}
Oct 30 05:47:03.830198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.829820    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" event={"ID":"61de2173-e6ea-4c2c-b3f5-c48200fef197","Type":"ContainerStarted","Data":"ef764ec7142d53f129a293c44b323dfc8d40c9a15ee67296e53e0c27b2e397ee"}
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867167    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867313    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867348    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867372    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867429    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dh6\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-kube-api-access-z8dh6\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867457    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867634    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867755    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.888204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.867821    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.892143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.892081    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" podStartSLOduration=1.892059449 podStartE2EDuration="1.892059449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:03.889584349 +0000 UTC m=+938.720621557" watchObservedRunningTime="2026-10-30 05:47:03.892059449 +0000 UTC m=+938.723096668"
Oct 30 05:47:03.956205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.952642    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.956205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.953323    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.956205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.954449    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.956205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.954525    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.956205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.955248    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.973035 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.968899    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:03.994211 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.993618    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 05:47:03.994211 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:03.993702    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 05:47:04.044823 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.044787    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:04.047242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.047210    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dh6\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-kube-api-access-z8dh6\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:04.047639 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.047602    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-c6w95\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:04.098206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.094614    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 05:47:04.103307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.103265    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 05:47:04.345289 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.345212    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:04.737652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.737618    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:04.739749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.739417    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 05:47:04.739749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.739475    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 05:47:04.935207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.916464    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"]
Oct 30 05:47:04.948214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.947358    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:04.948214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.947517    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:04.948214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.947612    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:05.006204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.990305    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 05:47:05.006204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:04.990386    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 05:47:05.067575 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.067533    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 05:47:05.067723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.067609    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 05:47:05.538805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.538767    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:05.702206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.701173    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 05:47:05.702206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.701252    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 05:47:05.904204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.900371    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" event={"ID":"6c4fc278-f251-4c6a-9c0d-1a185950d6ef","Type":"ContainerStarted","Data":"45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d"}
Oct 30 05:47:05.904204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.900436    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" event={"ID":"6c4fc278-f251-4c6a-9c0d-1a185950d6ef","Type":"ContainerStarted","Data":"f0c88eff0ff6095d1d9c82d38bdd838c9d0474d592586d398ac91fff0807e4d4"}
Oct 30 05:47:05.948999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.948938    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podStartSLOduration=2.948924422 podStartE2EDuration="2.948924422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:05.94670747 +0000 UTC m=+940.777744687" watchObservedRunningTime="2026-10-30 05:47:05.948924422 +0000 UTC m=+940.779961638"
Oct 30 05:47:05.976204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.963556    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 05:47:05.976204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:05.963612    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 05:47:06.077205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.067540    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 05:47:06.077205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.067601    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 05:47:06.351263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.350562    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:06.355224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.352503    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-c6w95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 05:47:06.355224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.352559    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:47:06.702205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.701618    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 05:47:06.702205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.701685    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 05:47:06.965267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.964615    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 05:47:06.965267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:06.964679    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 05:47:07.072206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.071290    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 05:47:07.072206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.071351    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 05:47:07.347256 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.346689    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-c6w95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 05:47:07.347256 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.346769    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:47:07.372217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.371850    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:07.705203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.704559    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 05:47:07.705203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.704622    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 05:47:07.889213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.889145    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:47:07.966650 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.966551    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 05:47:07.967264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:07.967231    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 05:47:08.023633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.023591    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:47:08.067570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.067533    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 05:47:08.067844 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.067813    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 05:47:08.276705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.276602    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:47:08.351888 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.351446    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-c6w95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 05:47:08.351888 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.351504    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:47:08.709381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.709329    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 05:47:08.709813 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.709762    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 05:47:08.965266 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.963587    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 05:47:08.965266 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:08.963650    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 05:47:09.068609 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.068569    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 05:47:09.069236 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.069204    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 05:47:09.345893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.345813    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-c6w95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 05:47:09.346337 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.346301    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:47:09.416350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.416272    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:09.701995 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.701557    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 05:47:09.701995 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.701619    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 05:47:09.977159 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.977074    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:09.978741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.978719    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:09.984498 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:09.984478    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 05:47:10.328237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.328104    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:10.329593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.329574    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:10.330270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.330254    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 05:47:10.345771 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.345744    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-c6w95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 05:47:10.345909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.345814    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:47:10.796194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.796153    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:10.809091 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.809066    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:10.833883 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.833852    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:10.834411 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.834390    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:10.835574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:10.835556    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:47:11.005216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:11.005157    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:47:11.154505 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:11.154475    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:11.346169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:11.346134    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-c6w95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 05:47:11.346589 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:11.346217    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:47:12.346528 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:12.346490    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-c6w95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 05:47:12.347014 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:12.346567    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:47:12.868114 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:12.868077    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:13.346584 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:13.346550    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-c6w95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 05:47:13.347162 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:13.347135    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:47:14.966939 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:14.966909    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:14.967461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:14.967390    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:14.968201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:14.968166    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:47:15.137575 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:15.137547    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:15.278705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:15.278619    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:15.615588 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:15.615499    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:16.610960 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:16.610930    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:17.152443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.152416    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"]
Oct 30 05:47:17.156704 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.156685    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.174151 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.174129    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fffaa8-openshift-default-dockercfg-fhmlj\""
Oct 30 05:47:17.191742 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.191719    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.191870 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.191768    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.191870 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.191794    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.191943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.191879    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.191943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.191926    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znd85\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-kube-api-access-znd85\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.192007 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.191949    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.192074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.192061    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.192120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.192105    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.192155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.192144    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.208606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.208579    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"]
Oct 30 05:47:17.293343 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293306    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znd85\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-kube-api-access-znd85\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293343 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293341    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293383    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293416    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293441    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293490    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293537    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293571    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293888 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293647    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293888 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293825    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293991 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293886    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.293991 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.293975    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.294090 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.294053    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.294531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.294510    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.296047 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.296026    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.296238 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.296223    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.315987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.315953    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.318060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.318039    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znd85\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-kube-api-access-znd85\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.470383 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.470302    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:17.791302 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.791270    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"]
Oct 30 05:47:17.792159 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:47:17.792078    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4d8c64_2eb5_4cca_bfa7_7edc6a0954dc.slice/crio-f21913a47db097a1b9b8e6b5b10a7c8c2b75e2c6497c661d99abaa89a0262620 WatchSource:0}: Error finding container f21913a47db097a1b9b8e6b5b10a7c8c2b75e2c6497c661d99abaa89a0262620: Status 404 returned error can't find the container with id f21913a47db097a1b9b8e6b5b10a7c8c2b75e2c6497c661d99abaa89a0262620
Oct 30 05:47:17.794659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.794629    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:17.794745 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.794691    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:17.794745 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.794736    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:17.897684 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:17.897636    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:18.056124 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:18.055949    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" event={"ID":"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc","Type":"ContainerStarted","Data":"c2abb731be8c04f5245eb52a7d09b9e70ce68b20c21834c04045ef999da8a609"}
Oct 30 05:47:18.056124 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:18.056029    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" event={"ID":"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc","Type":"ContainerStarted","Data":"f21913a47db097a1b9b8e6b5b10a7c8c2b75e2c6497c661d99abaa89a0262620"}
Oct 30 05:47:18.111979 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:18.111917    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" podStartSLOduration=1.111895431 podStartE2EDuration="1.111895431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:18.109478877 +0000 UTC m=+952.940516093" watchObservedRunningTime="2026-10-30 05:47:18.111895431 +0000 UTC m=+952.942932650"
Oct 30 05:47:18.475611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:18.475575    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:18.478141 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:18.478109    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 05:47:18.478396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:18.478369    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 05:47:18.501777 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:18.501752    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:19.475603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:19.475568    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 05:47:19.476238 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:19.476208    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 05:47:20.196258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:20.196207    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:20.473259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:20.472536    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 05:47:20.473259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:20.472617    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 05:47:21.478584 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:21.478548    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 05:47:21.479155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:21.479120    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 05:47:21.774256 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:21.771603    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:22.480567 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:22.480535    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 05:47:22.481024 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:22.481003    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 05:47:23.298381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:23.298349    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:23.477786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:23.477751    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 05:47:23.478070 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:23.478041    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 05:47:24.519712 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:24.519681    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:24.520633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:24.520610    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:24.522368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:24.522328    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:47:24.976872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:24.976781    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:26.942661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:26.942618    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:28.968293 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:28.968257    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:30.399797 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:30.399762    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:31.694559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:31.694523    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:31.902328 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:31.902295    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:32.375743 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:47:32.375702    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:47:33.934291 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:33.934255    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:35.541259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:35.541217    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:37.132141 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:37.132097    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:37.705722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:37.705683    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:39.065706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:39.065675    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:40.769478 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:40.769447    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:42.344854 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:42.344824    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:43.838331 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:43.838296    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:45.381877 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:45.381827    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:46.903504 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.903455    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"]
Oct 30 05:47:46.910501 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.910472    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.929299 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.929267    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"]
Oct 30 05:47:46.978615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.978570    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.978892 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.978866    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.979063 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.979045    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.979208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.979168    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.979347 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.979329    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.979467 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.979451    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.979579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.979565    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwt49\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-kube-api-access-mwt49\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.979707 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.979692    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:46.979848 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:46.979833    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.081582 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.081530    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.082011 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.081879    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.082239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.082219    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.082402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.082385    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.082550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.082532    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.082704 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.082685    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.082851 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.082833    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwt49\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-kube-api-access-mwt49\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.083020 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.082994    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.083220 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.083203    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.084648 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.084627    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.085902 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.084794    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.086055 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.085872    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.086231 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.086096    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.086775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.086756    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.089123 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.089052    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.099616 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.099591    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.108527 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.108477    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.113268 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.113215    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwt49\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-kube-api-access-mwt49\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.173517 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.173399    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:47.230574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.230530    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:47.592379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.592078    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"]
Oct 30 05:47:47.602311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.602278    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:47.602536 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.602496    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:47.602713 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:47.602684    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:47:48.307812 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:48.307777    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" event={"ID":"ef239f1c-5e8d-4450-aedd-ca14c332dd36","Type":"ContainerStarted","Data":"dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92"}
Oct 30 05:47:48.308242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:48.307831    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" event={"ID":"ef239f1c-5e8d-4450-aedd-ca14c332dd36","Type":"ContainerStarted","Data":"31ec5e93b5653173351965245e24db2d824cfc2bafad78edcefe7214c8749bdc"}
Oct 30 05:47:48.383866 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:48.383800    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podStartSLOduration=2.38378626 podStartE2EDuration="2.38378626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:48.370661305 +0000 UTC m=+983.201698526" watchObservedRunningTime="2026-10-30 05:47:48.38378626 +0000 UTC m=+983.214823479"
Oct 30 05:47:49.181588 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:49.178923    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:49.231141 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:49.231108    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:49.232997 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:49.232963    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 05:47:49.233143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:49.233035    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 05:47:50.230817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.230786    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 05:47:50.231234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.230834    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 05:47:50.888475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.888436    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"]
Oct 30 05:47:50.894574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.893998    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:50.910948 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.908598    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"]
Oct 30 05:47:50.915060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.915034    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"distinct-2-w6-64123f-kserve-self-signed-certs\""
Oct 30 05:47:50.979963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.979926    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tmp-dir\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:50.982505 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.982479    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-dshm\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:50.982767 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.982743    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8r7p\" (UniqueName: \"kubernetes.io/projected/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-kube-api-access-p8r7p\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:50.983635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.983612    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-model-cache\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:50.983843 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.983825    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-home\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:50.983984 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.983969    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tls-certs\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:50.984150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:50.983365    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:51.084946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.084902    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tmp-dir\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.085700 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.085676    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-dshm\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.086299 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.086277    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8r7p\" (UniqueName: \"kubernetes.io/projected/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-kube-api-access-p8r7p\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.086633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.086611    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-model-cache\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.086759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.085589    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tmp-dir\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.091390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.086927    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-model-cache\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.091551 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.087510    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-home\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.091709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.091692    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tls-certs\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.091893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.091756    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-dshm\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.091994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.088160    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-home\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.098376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.094730    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tls-certs\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.120164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.120132    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8r7p\" (UniqueName: \"kubernetes.io/projected/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-kube-api-access-p8r7p\") pod \"distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.137869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.137777    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:51.217947 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.217863    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:51.231960 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.231933    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 05:47:51.232405 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.231995    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 05:47:51.582830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:51.582140    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"]
Oct 30 05:47:51.585483 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:47:51.585456    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod573ce1fa_8d94_419b_99e4_ea42aa8ce51a.slice/crio-e8912529d4bc80635ddc913f5fc03571901bc5d0aef949cf5020363dba2c23e2 WatchSource:0}: Error finding container e8912529d4bc80635ddc913f5fc03571901bc5d0aef949cf5020363dba2c23e2: Status 404 returned error can't find the container with id e8912529d4bc80635ddc913f5fc03571901bc5d0aef949cf5020363dba2c23e2
Oct 30 05:47:52.231398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:52.231365    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 05:47:52.231577 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:52.231426    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 05:47:52.338468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:52.338432    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" event={"ID":"573ce1fa-8d94-419b-99e4-ea42aa8ce51a","Type":"ContainerStarted","Data":"c128d3ff292d0607c296acf33292c6198b870195f7d082d4d8c13d9402b0b6ac"}
Oct 30 05:47:52.338468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:52.338477    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" event={"ID":"573ce1fa-8d94-419b-99e4-ea42aa8ce51a","Type":"ContainerStarted","Data":"e8912529d4bc80635ddc913f5fc03571901bc5d0aef949cf5020363dba2c23e2"}
Oct 30 05:47:52.387832 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:52.387784    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podStartSLOduration=2.3877652830000002 podStartE2EDuration="2.387765283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:52.386251916 +0000 UTC m=+987.217289134" watchObservedRunningTime="2026-10-30 05:47:52.387765283 +0000 UTC m=+987.218802501"
Oct 30 05:47:53.037432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:53.037398    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:53.218191 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:53.218145    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:47:53.231101 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:53.231072    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 05:47:53.231274 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:53.231120    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 05:47:53.593857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:53.593813    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:54.231288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:54.231249    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 05:47:54.231501 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:54.231302    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 05:47:55.054365 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:55.054322    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:55.234884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:55.234854    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:55.235212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:55.235192    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:55.235775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:55.235757    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 05:47:56.818868 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:56.818832    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:58.262644 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:58.262600    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:59.664639 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:47:59.664595    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:01.218341 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:01.218308    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:48:01.231014 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:01.230987    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:48:01.266089 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:01.266038    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:01.415965 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:01.415935    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:48:02.375125 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:48:02.375087    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:48:02.718209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:02.718163    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:04.498068 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:04.498036    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:05.997675 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:05.997578    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:09.246210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:09.246147    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:11.154569 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:11.154535    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:12.851635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:12.851594    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:14.360331 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:14.360279    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:15.855371 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:15.855339    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:17.351679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:17.351644    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:18.769563 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:18.769523    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:22.055959 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:22.055928    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:24.277060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.277028    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"]
Oct 30 05:48:24.281102 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.281085    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.297349 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.297329    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:48:24.308939 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.308899    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 05:48:24.311647 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.311624    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"]
Oct 30 05:48:24.363634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.363605    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnml7\" (UniqueName: \"kubernetes.io/projected/2d0aedd7-d585-4784-b9b5-0d9c9512339c-kube-api-access-tnml7\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.363775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.363664    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.363831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.363808    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.464596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.464565    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.464739 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.464645    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.464784 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.464736    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnml7\" (UniqueName: \"kubernetes.io/projected/2d0aedd7-d585-4784-b9b5-0d9c9512339c-kube-api-access-tnml7\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.465315 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.465295    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.467111 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.467088    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.476466 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.476444    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:24.490669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.490632    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnml7\" (UniqueName: \"kubernetes.io/projected/2d0aedd7-d585-4784-b9b5-0d9c9512339c-kube-api-access-tnml7\") pod \"maas-api-revoke-sub-d9b15d0f-fsk2p\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.595941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.595865    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:24.900612 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:24.900583    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"]
Oct 30 05:48:24.902146 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:48:24.902111    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0aedd7_d585_4784_b9b5_0d9c9512339c.slice/crio-fe6dfac03babf01056e5cf52ccdb71ee38e340cb2389fc71995ac73e05bb5056 WatchSource:0}: Error finding container fe6dfac03babf01056e5cf52ccdb71ee38e340cb2389fc71995ac73e05bb5056: Status 404 returned error can't find the container with id fe6dfac03babf01056e5cf52ccdb71ee38e340cb2389fc71995ac73e05bb5056
Oct 30 05:48:25.601746 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:25.601652    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p" event={"ID":"2d0aedd7-d585-4784-b9b5-0d9c9512339c","Type":"ContainerStarted","Data":"269dfb650e5f95643fa7f2da5e734573dd6fcf7d95d36babe381f1a6a3def24d"}
Oct 30 05:48:25.601746 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:25.601721    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p" event={"ID":"2d0aedd7-d585-4784-b9b5-0d9c9512339c","Type":"ContainerStarted","Data":"fe6dfac03babf01056e5cf52ccdb71ee38e340cb2389fc71995ac73e05bb5056"}
Oct 30 05:48:25.643011 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:25.642955    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p" podStartSLOduration=1.64294098 podStartE2EDuration="1.64294098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:25.640137381 +0000 UTC m=+1020.471174600" watchObservedRunningTime="2026-10-30 05:48:25.64294098 +0000 UTC m=+1020.473978196"
Oct 30 05:48:25.923699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:25.923669    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:26.609713 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:26.609674    5250 generic.go:356] "Generic (PLEG): container finished" podID="2d0aedd7-d585-4784-b9b5-0d9c9512339c" containerID="269dfb650e5f95643fa7f2da5e734573dd6fcf7d95d36babe381f1a6a3def24d" exitCode=0
Oct 30 05:48:26.610173 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:26.609770    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p" event={"ID":"2d0aedd7-d585-4784-b9b5-0d9c9512339c","Type":"ContainerDied","Data":"269dfb650e5f95643fa7f2da5e734573dd6fcf7d95d36babe381f1a6a3def24d"}
Oct 30 05:48:27.567114 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.567081    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:27.774702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.774679    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:27.905401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.905374    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-xks-ca\") pod \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") "
Oct 30 05:48:27.905588 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.905435    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0aedd7-d585-4784-b9b5-0d9c9512339c-kube-api-access-tnml7\" (UniqueName: \"kubernetes.io/projected/2d0aedd7-d585-4784-b9b5-0d9c9512339c-kube-api-access-tnml7\") pod \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") "
Oct 30 05:48:27.905588 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.905557    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-openshift-ca\") pod \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\" (UID: \"2d0aedd7-d585-4784-b9b5-0d9c9512339c\") "
Oct 30 05:48:27.905895 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.905875    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-openshift-ca" pod "2d0aedd7-d585-4784-b9b5-0d9c9512339c" (UID: "2d0aedd7-d585-4784-b9b5-0d9c9512339c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:48:27.906120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.906077    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:27.907625 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.907603    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0aedd7-d585-4784-b9b5-0d9c9512339c-kube-api-access-tnml7" pod "2d0aedd7-d585-4784-b9b5-0d9c9512339c" (UID: "2d0aedd7-d585-4784-b9b5-0d9c9512339c"). InnerVolumeSpecName "kube-api-access-tnml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:48:27.907733 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:27.907688    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-xks-ca" pod "2d0aedd7-d585-4784-b9b5-0d9c9512339c" (UID: "2d0aedd7-d585-4784-b9b5-0d9c9512339c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:48:28.006759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:28.006730    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0aedd7-d585-4784-b9b5-0d9c9512339c-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:28.006759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:28.006756    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnml7\" (UniqueName: \"kubernetes.io/projected/2d0aedd7-d585-4784-b9b5-0d9c9512339c-kube-api-access-tnml7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:28.622954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:28.622922    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"
Oct 30 05:48:28.622954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:28.622941    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p" event={"ID":"2d0aedd7-d585-4784-b9b5-0d9c9512339c","Type":"ContainerDied","Data":"fe6dfac03babf01056e5cf52ccdb71ee38e340cb2389fc71995ac73e05bb5056"}
Oct 30 05:48:28.623150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:28.622991    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe6dfac03babf01056e5cf52ccdb71ee38e340cb2389fc71995ac73e05bb5056"
Oct 30 05:48:29.386884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:29.386853    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:30.973543 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:30.973488    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:31.724732 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.724703    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"]
Oct 30 05:48:31.725233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.725218    5250 cpu_manager.go:414] "containerMap: removing container" podUID="2d0aedd7-d585-4784-b9b5-0d9c9512339c" containerName="revoke-keys"
Oct 30 05:48:31.725282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.725235    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0aedd7-d585-4784-b9b5-0d9c9512339c" containerName="revoke-keys"
Oct 30 05:48:31.725372 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.725363    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0aedd7-d585-4784-b9b5-0d9c9512339c" containerName="revoke-keys"
Oct 30 05:48:31.729680 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.729650    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.763035 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.762996    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"]
Oct 30 05:48:31.768227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.768196    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"]
Oct 30 05:48:31.772991 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.772968    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.833484 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.833449    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"]
Oct 30 05:48:31.852494 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.852462    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.852679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.852508    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.852679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.852538    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.852787 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.852705    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.852832 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.852797    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.852877 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.852832    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.852960 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.852933    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.853026 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853008    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgkzn\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-kube-api-access-tgkzn\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.853101 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853083    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-956vz\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-kube-api-access-956vz\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.853158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853127    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b95a-7ae2-45f9-896a-5b9499810632-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.853227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853170    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.853227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853218    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.853385 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853365    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b95a-7ae2-45f9-896a-5b9499810632-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.853461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853441    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.853508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853480    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.853574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853557    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.853635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853602    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.853682 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.853640    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.865336 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.865310    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:31.955001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.954970    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955020    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955069    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955097    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.955195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955138    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.955195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955161    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgkzn\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-kube-api-access-tgkzn\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.955477 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955213    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-956vz\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-kube-api-access-956vz\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955477 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955248    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b95a-7ae2-45f9-896a-5b9499810632-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955477 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955292    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955477 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955325    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.955477 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955400    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b95a-7ae2-45f9-896a-5b9499810632-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955477 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955473    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.955788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955497    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955504    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.955788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955584    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.955788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955629    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.955788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955667    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.955788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955759    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.956100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955796    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.956100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955847    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.956100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955999    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.956100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.956096    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b95a-7ae2-45f9-896a-5b9499810632-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.956376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.955527    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.956376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.956286    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.956475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.956390    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.956475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.956402    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.956475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.956409    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.956708 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.956686    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.958228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.958165    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.958344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.958197    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.958639 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.958624    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b95a-7ae2-45f9-896a-5b9499810632-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:31.958767 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.958747    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.998615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.998555    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.999550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.999522    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgkzn\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-kube-api-access-tgkzn\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:31.999825 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:31.999805    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-956vz\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-kube-api-access-956vz\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:32.000859 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.000839    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:32.046131 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.046102    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:32.064882 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.064857    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"]
Oct 30 05:48:32.069117 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.069094    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.088153 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.088126    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:32.094235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.094148    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"]
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.157663    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.157784    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j294c\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-kube-api-access-j294c\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.157827    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.157889    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.157904    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.157941    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.157962    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.157987    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.158142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.158007    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259501    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259587 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259543    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259596    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259628    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259668    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259697    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259734    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259875    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j294c\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-kube-api-access-j294c\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259937    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.259994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.259954    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.260705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.260646    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.261039 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.260937    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.261039 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.261001    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.261478 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.261450    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.262677 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.262654    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.262981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.262958    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.282757 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.282736    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.283002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.282985    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j294c\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-kube-api-access-j294c\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.350936 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.350909    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"]
Oct 30 05:48:32.355950 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.355930    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.382726 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.382588    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"]
Oct 30 05:48:32.383923 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:48:32.383817    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:48:32.389162 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.387060    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"]
Oct 30 05:48:32.390506 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:48:32.390477    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3548b95a_7ae2_45f9_896a_5b9499810632.slice/crio-6dc8cc72fd9829f8a10cdd317ae0c46de5be0d0ad0cb6a7af1c2d4dd6e7a6712 WatchSource:0}: Error finding container 6dc8cc72fd9829f8a10cdd317ae0c46de5be0d0ad0cb6a7af1c2d4dd6e7a6712: Status 404 returned error can't find the container with id 6dc8cc72fd9829f8a10cdd317ae0c46de5be0d0ad0cb6a7af1c2d4dd6e7a6712
Oct 30 05:48:32.393644 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.393606    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:32.393768 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.393705    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:32.393768 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.393750    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:32.404962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.404937    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:32.439049 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.439022    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"]
Oct 30 05:48:32.439827 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:48:32.439798    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5854ee_8d3d_4429_9eb4_f1b3d11861f2.slice/crio-bb328b6b1ba8381509e3339f89153cb86c56c4473fc5c88fa2477bba156452d8 WatchSource:0}: Error finding container bb328b6b1ba8381509e3339f89153cb86c56c4473fc5c88fa2477bba156452d8: Status 404 returned error can't find the container with id bb328b6b1ba8381509e3339f89153cb86c56c4473fc5c88fa2477bba156452d8
Oct 30 05:48:32.442366 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.442329    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:32.442463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.442412    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:32.442463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.442450    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:32.465912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.465883    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.466022 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.465950    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh6j7\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-kube-api-access-jh6j7\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.466022 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.466013    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.466124 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.466107    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.466220 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.466152    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.466220 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.466212    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.466340 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.466248    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096884b-f166-45ff-9590-81d96cba7d2c-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.466340 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.466295    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096884b-f166-45ff-9590-81d96cba7d2c-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.466493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.466337    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.500131 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.499933    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.567790    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096884b-f166-45ff-9590-81d96cba7d2c-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.567835    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.567884    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.567917    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh6j7\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-kube-api-access-jh6j7\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.567960    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.568020    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.568047    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.568073    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.568093    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096884b-f166-45ff-9590-81d96cba7d2c-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569164 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.568850    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096884b-f166-45ff-9590-81d96cba7d2c-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.569246    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.569934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.569820    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.570407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.570078    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.571160 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.571138    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.573659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.573616    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096884b-f166-45ff-9590-81d96cba7d2c-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.573906 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.573882    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.591924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.591899    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.592162 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.591939    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh6j7\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-kube-api-access-jh6j7\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.652330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.651608    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" event={"ID":"3548b95a-7ae2-45f9-896a-5b9499810632","Type":"ContainerStarted","Data":"795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986"}
Oct 30 05:48:32.652330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.651671    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" event={"ID":"3548b95a-7ae2-45f9-896a-5b9499810632","Type":"ContainerStarted","Data":"6dc8cc72fd9829f8a10cdd317ae0c46de5be0d0ad0cb6a7af1c2d4dd6e7a6712"}
Oct 30 05:48:32.653615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.653549    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" event={"ID":"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2","Type":"ContainerStarted","Data":"aa20b2223e0ff59efc9bc28f9fa9e7b0ff63ead7c54738ef7215e3287a4758da"}
Oct 30 05:48:32.653615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.653592    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" event={"ID":"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2","Type":"ContainerStarted","Data":"bb328b6b1ba8381509e3339f89153cb86c56c4473fc5c88fa2477bba156452d8"}
Oct 30 05:48:32.672061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.672038    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:32.702735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.702678    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podStartSLOduration=1.702659176 podStartE2EDuration="1.702659176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:32.696974204 +0000 UTC m=+1027.528011423" watchObservedRunningTime="2026-10-30 05:48:32.702659176 +0000 UTC m=+1027.533696393"
Oct 30 05:48:32.757633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.757096    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podStartSLOduration=1.75704546 podStartE2EDuration="1.75704546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:32.749083736 +0000 UTC m=+1027.580120965" watchObservedRunningTime="2026-10-30 05:48:32.75704546 +0000 UTC m=+1027.588082678"
Oct 30 05:48:32.759104 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.758818    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"]
Oct 30 05:48:32.768500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.762473    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:32.768500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.762557    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:32.768500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:32.762597    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:33.004170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.004129    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"]
Oct 30 05:48:33.026088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.026052    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:33.026251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.026121    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:33.026251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.026150    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:33.048047 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.046569    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:33.048498 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.048468    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 05:48:33.048618 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.048535    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 05:48:33.090538 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.090388    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:33.091210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.090802    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 05:48:33.091210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.090856    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 05:48:33.690612 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.690573    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" event={"ID":"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b","Type":"ContainerStarted","Data":"de5ad4386b16dae4df5a92d4853ef57fe17998d4eee491c386e4991ecbfd6283"}
Oct 30 05:48:33.690883 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.690861    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" event={"ID":"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b","Type":"ContainerStarted","Data":"6d04c7226ec8bf1dfdb1ee78ed98d89487c42e11606bb228e74162fa2feb6add"}
Oct 30 05:48:33.694244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.694215    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" event={"ID":"b096884b-f166-45ff-9590-81d96cba7d2c","Type":"ContainerStarted","Data":"ec0ce2d400890f81b8ca279a12faadc381d8c14825a1c06e48f37061fd074a9a"}
Oct 30 05:48:33.694521 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.694504    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" event={"ID":"b096884b-f166-45ff-9590-81d96cba7d2c","Type":"ContainerStarted","Data":"700461a81bc2a2640c7e2684c378bb76d0cd9f1ca32d5ad59a5f591bd93a1458"}
Oct 30 05:48:33.749368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.749318    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podStartSLOduration=1.749304563 podStartE2EDuration="1.749304563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:33.748421074 +0000 UTC m=+1028.579458292" watchObservedRunningTime="2026-10-30 05:48:33.749304563 +0000 UTC m=+1028.580341780"
Oct 30 05:48:33.829632 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.829598    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:33.991101 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:33.990980    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:34.051226 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.051191    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 05:48:34.051909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.051855    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 05:48:34.095762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.095635    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 05:48:34.095762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.095699    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 05:48:34.493207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.484570    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:34.522206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.514791    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:48:34.522206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.514852    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:48:34.619491 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.619437    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:34.692537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.692503    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:34.720987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.720945    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:48:34.721275 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:34.721246    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:48:35.047017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.046580    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 05:48:35.047017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.046642    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 05:48:35.076074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.076015    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:35.098415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.098285    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 05:48:35.098415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.098350    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 05:48:35.346388 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.346286    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 05:48:35.421661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.421621    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:48:35.421992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.421951    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:48:35.697264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.683556    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:48:35.697264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.683617    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:48:35.838204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:35.837754    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:36.055380 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.055273    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 05:48:36.087212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.078899    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:36.133206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.101475    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:48:36.133206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.102468    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:48:36.167206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.141685    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 05:48:36.245024 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.244988    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 05:48:36.245642 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.243290    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 05:48:36.259746 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.259716    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 05:48:36.267300 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.267259    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 05:48:36.297204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.296508    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podStartSLOduration=4.296485266 podStartE2EDuration="4.296485266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:33.804533167 +0000 UTC m=+1028.635570385" watchObservedRunningTime="2026-10-30 05:48:36.296485266 +0000 UTC m=+1031.127522484"
Oct 30 05:48:36.373392 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.373287    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:36.439688 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.374366    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:36.440232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.440208    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:48:36.449288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.415040    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:36.457317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.414996    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:48:36.457550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.457507    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:48:36.564211 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.555910    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 05:48:36.769311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.761593    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:48:36.769311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.761658    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:48:36.898296 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.898249    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:48:36.900765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:36.900739    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:48:37.061908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.061811    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 05:48:37.061908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.061872    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 05:48:37.062317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.062282    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:37.063553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.063533    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:48:37.105565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.083639    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:48:37.106161 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.105624    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:37.106161 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.095578    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:37.113596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.095066    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 05:48:37.113884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.113851    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 05:48:37.167959 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.167918    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:37.223246 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.191345    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:48:37.383617 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.383308    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 05:48:37.387207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.385227    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:48:37.443455 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.443387    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:37.443886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.443847    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:37.509216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.444844    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:48:37.630205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.614071    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:48:37.649699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.649597    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:48:37.824362 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.824325    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:48:37.907579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.907458    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:48:37.938311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.906253    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:48:37.962605 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:37.930854    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:38.072442 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.072406    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 05:48:38.072725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.072697    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 05:48:38.175387 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.175284    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:48:38.179271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.179242    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:48:38.179520 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.179504    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:48:38.179656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.179645    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:48:38.179764 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.179755    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:48:38.217502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.217446    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:38.218107 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.218086    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:48:38.306170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.306133    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 05:48:38.306500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.306466    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 05:48:38.358203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.353233    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:48:38.393206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.392800    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:38.425287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.425253    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:48:38.425625 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.425549    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:48:38.445461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.445415    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:38.728771 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.728678    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:48:38.729016 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.728988    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:48:38.835203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.826269    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:48:38.875220 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.875150    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:38.931258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.915897    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:38.973206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.972715    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:38.973418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.973200    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:38.973418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.973278    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:48:38.973666 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:38.973642    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:48:39.001890 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.001673    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:39.114457 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.114424    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 05:48:39.133280 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.133145    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 05:48:39.135231 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.115376    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 05:48:39.135471 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.135444    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 05:48:39.189214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.186809    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:39.243207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.220272    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:39.354347 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.354251    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:39.396860 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.396804    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:39.405543 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.405511    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:48:39.405766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.405742    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:48:39.459446 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.459389    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:39.501205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.500666    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:48:39.501205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.500732    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:48:39.564214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.552458    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:48:39.734318 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.683125    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:48:39.734318 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.683199    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:48:39.752749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.752637    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:48:39.937741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:39.937695    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:40.068262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.047690    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:48:40.068262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.064503    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:40.068262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.065574    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:48:40.124203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.097135    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 05:48:40.124203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.097225    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 05:48:40.124203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.119337    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 05:48:40.124203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.119391    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 05:48:40.145124 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.145085    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:48:40.147264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.146158    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:48:40.232413 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.232382    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:48:40.235672 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.235642    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:48:40.306136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.306106    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:40.414169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.414132    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:48:40.414523 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.414458    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:48:40.677266 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.673201    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:48:40.677266 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:40.673263    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:48:41.399152 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.399117    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:41.400571 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.400551    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:41.405248 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.405218    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 05:48:41.405775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.405755    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:48:41.405993 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.405951    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:48:41.410882 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.410860    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"]
Oct 30 05:48:41.416010 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.415992    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.435017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.432513    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 05:48:41.435017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.434331    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:48:41.440593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.439899    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.440593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.439976    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhwp\" (UniqueName: \"kubernetes.io/projected/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-kube-api-access-bwhwp\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.440593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.440160    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.445052 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.444873    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"]
Oct 30 05:48:41.495327 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.489424    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:41.495327 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.490718    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:41.499153 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.497816    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 05:48:41.541727 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.541697    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.542102 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.542077    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhwp\" (UniqueName: \"kubernetes.io/projected/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-kube-api-access-bwhwp\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.542619 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.542598    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.544450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.544429    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.551053 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.551034    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.568648 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.568623    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhwp\" (UniqueName: \"kubernetes.io/projected/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-kube-api-access-bwhwp\") pod \"maas-api-revoke-sub-3647e369-7rz65\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:41.673398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.673309    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:48:41.673398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.673370    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:48:41.744628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:41.744593    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:42.077662 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.077632    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"]
Oct 30 05:48:42.079283 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:48:42.079252    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fe9d4f_6b5b_4bb8_9eff_d8cf6a2ad15d.slice/crio-ee9be9a9ac7e6ba8065bc0ec1528166d2be3c176114341379a92c1e64372e12a WatchSource:0}: Error finding container ee9be9a9ac7e6ba8065bc0ec1528166d2be3c176114341379a92c1e64372e12a: Status 404 returned error can't find the container with id ee9be9a9ac7e6ba8065bc0ec1528166d2be3c176114341379a92c1e64372e12a
Oct 30 05:48:42.410539 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.410512    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:42.411066 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.410996    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:42.411728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.411709    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 05:48:42.463821 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.463780    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65" event={"ID":"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d","Type":"ContainerStarted","Data":"35957b6bb5c1877c5294a2ac0b5a313d7fa2671d4e6120731955e6565dbccad7"}
Oct 30 05:48:42.463961 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.463837    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65" event={"ID":"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d","Type":"ContainerStarted","Data":"ee9be9a9ac7e6ba8065bc0ec1528166d2be3c176114341379a92c1e64372e12a"}
Oct 30 05:48:42.541351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.541280    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65" podStartSLOduration=1.541261397 podStartE2EDuration="1.541261397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:42.538568486 +0000 UTC m=+1037.369605723" watchObservedRunningTime="2026-10-30 05:48:42.541261397 +0000 UTC m=+1037.372298616"
Oct 30 05:48:42.677463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.677388    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:42.677676 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.677659    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:42.678485 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:42.678468    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 05:48:43.004818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:43.004732    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:43.471230 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:43.471168    5250 generic.go:356] "Generic (PLEG): container finished" podID="46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d" containerID="35957b6bb5c1877c5294a2ac0b5a313d7fa2671d4e6120731955e6565dbccad7" exitCode=0
Oct 30 05:48:43.471690 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:43.471231    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65" event={"ID":"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d","Type":"ContainerDied","Data":"35957b6bb5c1877c5294a2ac0b5a313d7fa2671d4e6120731955e6565dbccad7"}
Oct 30 05:48:44.619502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.619479    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:44.693585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.693555    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-kube-api-access-bwhwp\" (UniqueName: \"kubernetes.io/projected/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-kube-api-access-bwhwp\") pod \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") "
Oct 30 05:48:44.693763 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.693696    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-xks-ca\") pod \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") "
Oct 30 05:48:44.693763 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.693752    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-openshift-ca\") pod \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\" (UID: \"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d\") "
Oct 30 05:48:44.694257 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.694230    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-openshift-ca" pod "46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d" (UID: "46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:48:44.694457 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.694427    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:44.695954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.695929    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-kube-api-access-bwhwp" pod "46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d" (UID: "46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d"). InnerVolumeSpecName "kube-api-access-bwhwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:48:44.696399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.696382    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-xks-ca" pod "46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d" (UID: "46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:48:44.774417 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.774339    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:44.797576 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.797538    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwhwp\" (UniqueName: \"kubernetes.io/projected/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-kube-api-access-bwhwp\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:44.797576 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:44.797580    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:45.486441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:45.486407    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"
Oct 30 05:48:45.486603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:45.486433    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65" event={"ID":"46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d","Type":"ContainerDied","Data":"ee9be9a9ac7e6ba8065bc0ec1528166d2be3c176114341379a92c1e64372e12a"}
Oct 30 05:48:45.486603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:45.486502    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9be9a9ac7e6ba8065bc0ec1528166d2be3c176114341379a92c1e64372e12a"
Oct 30 05:48:47.040718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.039775    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"]
Oct 30 05:48:47.040718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.040690    5250 cpu_manager.go:414] "containerMap: removing container" podUID="46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d" containerName="revoke-keys"
Oct 30 05:48:47.040718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.040713    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d" containerName="revoke-keys"
Oct 30 05:48:47.041521 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.040846    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d" containerName="revoke-keys"
Oct 30 05:48:47.053489 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.053460    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.083665 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.083631    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"]
Oct 30 05:48:47.313269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.305459    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.323389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.323338    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.336207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.327299    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a61ae66-7883-48bb-b2d2-0c65deb92338-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.337329 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.337291    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.337541 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.337523    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb9bz\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-kube-api-access-lb9bz\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.337685 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.337668    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.337802 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.337787    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.337929 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.337914    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.338063 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.338048    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.409218 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.407689    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:47.431375 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.409624    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 05:48:47.460221 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.444019    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.464203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.460585    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.474449 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.474422    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a61ae66-7883-48bb-b2d2-0c65deb92338-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.486506 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.486474    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.515205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.490629    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb9bz\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-kube-api-access-lb9bz\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.515561 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.515530    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.515707 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.515687    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.522317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.515843    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.532831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.522556    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.555435 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.468636    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:47.581264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.468272    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:47.629452 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.478048    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a61ae66-7883-48bb-b2d2-0c65deb92338-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.654640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.488604    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.658273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.527612    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.702205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.527849    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.702205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.528080    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.702205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.445297    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:47.702205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.629568    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb9bz\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-kube-api-access-lb9bz\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.702205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.693317    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.817447 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.817411    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.819130 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.817898    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:47.841987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.841268    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:47.975355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:47.975292    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.028729 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.028659    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:48.054252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.053311    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.221206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.218737    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.246324 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.245920    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.414205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.409436    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.414205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.410168    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:48:48.414205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.411351    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.414205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.411529    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:48:48.638407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.601115    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.638407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.603360    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:48:48.638407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.628728    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:48:48.638407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.628793    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:48:48.843329 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.836775    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 05:48:48.882201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.868276    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.882201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.869239    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:48.898287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.882658    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:48:48.898287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:48.882751    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:48:49.025207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.020937    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:48:49.074207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.029301    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:49.074207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.029509    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:48:49.240379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.201028    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 05:48:49.240379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.237232    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:48:49.240379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.238084    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:48:49.256207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.252947    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:49.256422 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.256373    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:48:49.256499 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.256456    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:48:49.524210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.431391    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:49.524210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.433784    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:49.773913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.736408    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 05:48:49.773913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.736799    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:48:49.773913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.837274    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:49.773913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:49.873293    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.069634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.028317    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:48:50.138002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.137510    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:50.138002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.137642    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:50.138002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.137692    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:48:50.143863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.143769    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"]
Oct 30 05:48:50.236322 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.236265    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 05:48:50.260554 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.258094    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.260554 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.259864    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.260554 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.260003    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.361647 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.361524    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.439826 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.439778    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.521212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.517933    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" event={"ID":"9a61ae66-7883-48bb-b2d2-0c65deb92338","Type":"ContainerStarted","Data":"cd0ed9cdbf2f4e8bca4d42bea214af48315ae08a972b6c49b04cbed970526ba9"}
Oct 30 05:48:50.544285 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.534948    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.762239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.754802    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.852425 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.852380    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 05:48:50.900284 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.862318    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.920206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.904324    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:50.928225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:50.925987    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:51.032900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.029858    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:51.032900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.032450    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:48:51.263856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.263797    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:51.273222 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.270112    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:48:51.283228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.280608    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:48:51.283228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.281483    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:48:51.283228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.281786    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:48:51.343253 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.343222    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:48:51.343429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.343282    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:48:51.343429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.343311    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:48:51.343429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.343341    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:48:51.343429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.343375    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:48:51.344949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.344915    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:48:51.368069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.368042    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:48:51.531715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.531618    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" event={"ID":"9a61ae66-7883-48bb-b2d2-0c65deb92338","Type":"ContainerStarted","Data":"51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df"}
Oct 30 05:48:51.684740 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:51.684682    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podStartSLOduration=5.68466369 podStartE2EDuration="5.68466369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:51.683810748 +0000 UTC m=+1046.514847967" watchObservedRunningTime="2026-10-30 05:48:51.68466369 +0000 UTC m=+1046.515700908"
Oct 30 05:48:52.029563 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:52.029529    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:52.030979 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:52.030951    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:48:52.031109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:52.031003    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:48:53.029108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.029075    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:48:53.029536 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.029127    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:48:53.750305 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.750276    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"]
Oct 30 05:48:53.754552 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.754532    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:53.792649 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.792548    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 05:48:53.792649 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.792619    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:48:53.796524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.796497    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"]
Oct 30 05:48:53.899741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.899710    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:53.899889 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.899845    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfnv\" (UniqueName: \"kubernetes.io/projected/7e4e35f9-3cda-4b70-a615-197aeb09fb32-kube-api-access-wwfnv\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:53.899949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:53.899887    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:54.000809 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.000729    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfnv\" (UniqueName: \"kubernetes.io/projected/7e4e35f9-3cda-4b70-a615-197aeb09fb32-kube-api-access-wwfnv\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:54.000809 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.000780    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:54.001051 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.001028    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:54.001662 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.001582    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:54.003312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.003290    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:54.024363 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.024340    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfnv\" (UniqueName: \"kubernetes.io/projected/7e4e35f9-3cda-4b70-a615-197aeb09fb32-kube-api-access-wwfnv\") pod \"maas-api-revoke-sub-916f3685-mpwxd\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:54.029198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.029154    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:48:54.029602 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.029227    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:48:54.113352 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.113308    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:54.442908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.442880    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"]
Oct 30 05:48:54.446248 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:48:54.446212    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4e35f9_3cda_4b70_a615_197aeb09fb32.slice/crio-279f1e9301598881faab274c65a2c187f87fb3e2616b7b2379603256c7ec6143 WatchSource:0}: Error finding container 279f1e9301598881faab274c65a2c187f87fb3e2616b7b2379603256c7ec6143: Status 404 returned error can't find the container with id 279f1e9301598881faab274c65a2c187f87fb3e2616b7b2379603256c7ec6143
Oct 30 05:48:54.551959 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:54.551924    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd" event={"ID":"7e4e35f9-3cda-4b70-a615-197aeb09fb32","Type":"ContainerStarted","Data":"279f1e9301598881faab274c65a2c187f87fb3e2616b7b2379603256c7ec6143"}
Oct 30 05:48:55.024423 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:55.024392    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:55.030068 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:55.030042    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:48:55.030428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:55.030086    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:48:55.569214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:55.567467    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd" event={"ID":"7e4e35f9-3cda-4b70-a615-197aeb09fb32","Type":"ContainerStarted","Data":"62b2505d60b75527eff17d5fcd4522357edb391a9a0dd30286b52b3a262e2584"}
Oct 30 05:48:55.612122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:55.612076    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd" podStartSLOduration=2.612063871 podStartE2EDuration="2.612063871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:55.611690362 +0000 UTC m=+1050.442727582" watchObservedRunningTime="2026-10-30 05:48:55.612063871 +0000 UTC m=+1050.443101088"
Oct 30 05:48:56.029468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:56.029428    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:48:56.029642 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:56.029492    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:48:56.587800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:56.587762    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:56.593309 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:56.593283    5250 generic.go:356] "Generic (PLEG): container finished" podID="7e4e35f9-3cda-4b70-a615-197aeb09fb32" containerID="62b2505d60b75527eff17d5fcd4522357edb391a9a0dd30286b52b3a262e2584" exitCode=0
Oct 30 05:48:56.593457 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:56.593371    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd" event={"ID":"7e4e35f9-3cda-4b70-a615-197aeb09fb32","Type":"ContainerDied","Data":"62b2505d60b75527eff17d5fcd4522357edb391a9a0dd30286b52b3a262e2584"}
Oct 30 05:48:57.029761 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.029725    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:48:57.029957 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.029785    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:48:57.228057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.227994    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:57.809820 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.809794    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:57.858689 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.858603    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4e35f9-3cda-4b70-a615-197aeb09fb32-kube-api-access-wwfnv\" (UniqueName: \"kubernetes.io/projected/7e4e35f9-3cda-4b70-a615-197aeb09fb32-kube-api-access-wwfnv\") pod \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") "
Oct 30 05:48:57.858888 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.858773    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-openshift-ca\") pod \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") "
Oct 30 05:48:57.858888 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.858838    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-xks-ca\") pod \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\" (UID: \"7e4e35f9-3cda-4b70-a615-197aeb09fb32\") "
Oct 30 05:48:57.860511 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.860480    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-openshift-ca" pod "7e4e35f9-3cda-4b70-a615-197aeb09fb32" (UID: "7e4e35f9-3cda-4b70-a615-197aeb09fb32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:48:57.862411 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.862385    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-xks-ca" pod "7e4e35f9-3cda-4b70-a615-197aeb09fb32" (UID: "7e4e35f9-3cda-4b70-a615-197aeb09fb32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:48:57.862660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.862620    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4e35f9-3cda-4b70-a615-197aeb09fb32-kube-api-access-wwfnv" pod "7e4e35f9-3cda-4b70-a615-197aeb09fb32" (UID: "7e4e35f9-3cda-4b70-a615-197aeb09fb32"). InnerVolumeSpecName "kube-api-access-wwfnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:48:57.960392 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.960345    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfnv\" (UniqueName: \"kubernetes.io/projected/7e4e35f9-3cda-4b70-a615-197aeb09fb32-kube-api-access-wwfnv\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:57.960392 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.960388    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:57.960392 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:57.960398    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4e35f9-3cda-4b70-a615-197aeb09fb32-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:48:58.029008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:58.028974    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:48:58.029225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:58.029026    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:48:58.609732 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:58.609699    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"
Oct 30 05:48:58.609913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:58.609733    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd" event={"ID":"7e4e35f9-3cda-4b70-a615-197aeb09fb32","Type":"ContainerDied","Data":"279f1e9301598881faab274c65a2c187f87fb3e2616b7b2379603256c7ec6143"}
Oct 30 05:48:58.609913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:58.609793    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="279f1e9301598881faab274c65a2c187f87fb3e2616b7b2379603256c7ec6143"
Oct 30 05:48:58.899171 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:58.899112    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:59.035930 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:59.035899    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:59.036333 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:59.036306    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:48:59.037268 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:48:59.037251    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:49:00.695071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:00.695042    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:02.377278 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:49:02.377237    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:49:02.424035 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.424001    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:02.643046 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.643015    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"]
Oct 30 05:49:02.643673 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.643650    5250 cpu_manager.go:414] "containerMap: removing container" podUID="7e4e35f9-3cda-4b70-a615-197aeb09fb32" containerName="revoke-keys"
Oct 30 05:49:02.643673 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.643673    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4e35f9-3cda-4b70-a615-197aeb09fb32" containerName="revoke-keys"
Oct 30 05:49:02.643864 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.643852    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4e35f9-3cda-4b70-a615-197aeb09fb32" containerName="revoke-keys"
Oct 30 05:49:02.647850 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.647832    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.685524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.685489    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"]
Oct 30 05:49:02.719001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.718966    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.719210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.719133    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.719301 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.719226    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b436d1e3-a390-48b6-8a1c-493850bf5817-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.719301 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.719273    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.719380 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.719359    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.719414 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.719385    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.719452 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.719415    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.719558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.719539    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kgbp\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-kube-api-access-2kgbp\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.719697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.719679    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821141 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821104    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821155    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821217    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821254    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b436d1e3-a390-48b6-8a1c-493850bf5817-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821306    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821358    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821378    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821412    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821499    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgbp\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-kube-api-access-2kgbp\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821594    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821635    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821724    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.821894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.821843    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.822279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.822261    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b436d1e3-a390-48b6-8a1c-493850bf5817-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.823629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.823608    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.823903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.823886    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.844261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.844233    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.844370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.844326    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kgbp\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-kube-api-access-2kgbp\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:02.963321 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:02.963251    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:03.285270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.285243    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"]
Oct 30 05:49:03.285402 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:49:03.285344    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb436d1e3_a390_48b6_8a1c_493850bf5817.slice/crio-7a4fb5e3cd4f30b0fc9e216c03993ed76c316c27cc9a31346f874127ca32a170 WatchSource:0}: Error finding container 7a4fb5e3cd4f30b0fc9e216c03993ed76c316c27cc9a31346f874127ca32a170: Status 404 returned error can't find the container with id 7a4fb5e3cd4f30b0fc9e216c03993ed76c316c27cc9a31346f874127ca32a170
Oct 30 05:49:03.287854 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.287822    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:49:03.287955 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.287889    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:49:03.287955 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.287919    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:49:03.647108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.647068    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" event={"ID":"b436d1e3-a390-48b6-8a1c-493850bf5817","Type":"ContainerStarted","Data":"0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e"}
Oct 30 05:49:03.647500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.647126    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" event={"ID":"b436d1e3-a390-48b6-8a1c-493850bf5817","Type":"ContainerStarted","Data":"7a4fb5e3cd4f30b0fc9e216c03993ed76c316c27cc9a31346f874127ca32a170"}
Oct 30 05:49:03.702343 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.702293    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" podStartSLOduration=1.702278685 podStartE2EDuration="1.702278685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:03.700789582 +0000 UTC m=+1058.531826800" watchObservedRunningTime="2026-10-30 05:49:03.702278685 +0000 UTC m=+1058.533315959"
Oct 30 05:49:03.969449 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.969312    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:03.980212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:03.978423    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:04.978203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:04.971278    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:49:04.978203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:04.971334    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:49:05.882216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:05.879352    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:05.971209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:05.966613    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:49:05.971209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:05.966671    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:49:06.980658 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:06.980618    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:49:06.981247 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:06.980689    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:49:07.477739 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:07.477703    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:07.980239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:07.973102    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:49:07.980239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:07.973162    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:49:08.980344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:08.980309    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:49:08.980740 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:08.980384    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:49:09.275677 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:09.275643    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:09.969573 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:09.969544    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:09.970067 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:09.970044    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:09.970925 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:09.970902    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 05:49:10.688772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:10.688740    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:12.384406 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:12.384340    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:15.265210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.264272    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:49:15.277237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.275518    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:49:15.348204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.345893    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:15.348204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.347259    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 05:49:15.348204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.347982    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 05:49:15.370275 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.370233    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:15.439399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.439330    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:15.440108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.440074    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:15.750705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.750566    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:15.941914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:15.941879    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:16.268338 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.268278    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:16.269705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.269680    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:49:16.276260 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.276230    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:49:16.276772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.276752    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:49:16.349306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.349264    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:16.349894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.349861    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:49:16.350337 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.350312    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 05:49:16.351060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.351041    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:49:16.351252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.351238    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:49:16.351428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.351415    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:49:16.378433 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.370993    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:16.378433 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.371375    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:49:16.723332 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.723287    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 05:49:16.783262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.738513    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 05:49:16.783262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.738914    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:16.783262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.739242    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:49:16.783262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.739300    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:49:16.804205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.787427    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:16.804205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.787805    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:49:16.917205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:16.897293    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:17.064254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.037827    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:49:17.271351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.271306    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:17.316841 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.316733    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:49:17.359849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.359810    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:17.390728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.369579    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 05:49:17.390728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.369860    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:49:17.798065 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.798010    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:17.798751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.798719    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 05:49:17.888478 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.876869    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:17.908560 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.907660    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:49:17.911793 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.911770    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:49:17.969094 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:17.969067    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:49:18.087011 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.086918    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:49:18.100705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.100677    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:49:18.141882 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.131446    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:49:18.265227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.263538    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:49:18.284271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.277483    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:18.355550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.355456    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:49:18.370382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.370337    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:18.570203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.561432    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:49:18.626755 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.618914    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:49:18.713425 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.711968    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:49:18.753986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:18.753964    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:49:21.309201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.305762    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"]
Oct 30 05:49:21.330511 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.330468    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"]
Oct 30 05:49:21.330676 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.330590    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.356691 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.356665    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"openshift-service-ca.crt\""
Oct 30 05:49:21.356691 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.356683    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"kube-root-ca.crt\""
Oct 30 05:49:21.356877 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.356682    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"distinct-w2-e7a506-kserve-self-signed-certs\""
Oct 30 05:49:21.356877 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.356719    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"default-dockercfg-mgnkm\""
Oct 30 05:49:21.405877 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.405842    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:21.441146 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.441109    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-home\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.441381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.441301    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2c57\" (UniqueName: \"kubernetes.io/projected/0c92033e-9264-4ae8-b0b6-779c547dba3c-kube-api-access-k2c57\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.441487 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.441457    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-dshm\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.441703 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.441676    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c92033e-9264-4ae8-b0b6-779c547dba3c-tls-certs\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.441863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.441846    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-model-cache\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.441932 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.441919    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-tmp-dir\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.543049 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.543003    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-dshm\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.543278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.543063    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c92033e-9264-4ae8-b0b6-779c547dba3c-tls-certs\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.543278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.543168    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-model-cache\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.543407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.543279    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-tmp-dir\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.543407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.543337    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-home\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.543531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.543433    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2c57\" (UniqueName: \"kubernetes.io/projected/0c92033e-9264-4ae8-b0b6-779c547dba3c-kube-api-access-k2c57\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.543843 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.543811    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-model-cache\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.544119 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.544093    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-tmp-dir\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.544218 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.544153    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-home\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.546559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.546536    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-dshm\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.546937 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.546918    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c92033e-9264-4ae8-b0b6-779c547dba3c-tls-certs\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.568074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.567971    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2c57\" (UniqueName: \"kubernetes.io/projected/0c92033e-9264-4ae8-b0b6-779c547dba3c-kube-api-access-k2c57\") pod \"distinct-w2-e7a506-kserve-554f6d58f5-rt2s2\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.646859 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.646815    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:21.979298 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:49:21.979257    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c92033e_9264_4ae8_b0b6_779c547dba3c.slice/crio-ff5111655cfa130c792ca6f0622b06ca1cc40382271239c627324c724cd398bc WatchSource:0}: Error finding container ff5111655cfa130c792ca6f0622b06ca1cc40382271239c627324c724cd398bc: Status 404 returned error can't find the container with id ff5111655cfa130c792ca6f0622b06ca1cc40382271239c627324c724cd398bc
Oct 30 05:49:21.981059 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:21.981027    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"]
Oct 30 05:49:22.219745 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:22.219709    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" event={"ID":"0c92033e-9264-4ae8-b0b6-779c547dba3c","Type":"ContainerStarted","Data":"1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1"}
Oct 30 05:49:22.219910 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:22.219758    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" event={"ID":"0c92033e-9264-4ae8-b0b6-779c547dba3c","Type":"ContainerStarted","Data":"ff5111655cfa130c792ca6f0622b06ca1cc40382271239c627324c724cd398bc"}
Oct 30 05:49:22.262494 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:22.262388    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podStartSLOduration=1.262368512 podStartE2EDuration="1.262368512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:22.260040777 +0000 UTC m=+1077.091077994" watchObservedRunningTime="2026-10-30 05:49:22.262368512 +0000 UTC m=+1077.093405730"
Oct 30 05:49:22.427990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:22.427959    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:22.647199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:22.647154    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:24.097145 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:24.097059    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:25.609545 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:25.609512    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:27.287578 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:27.287529    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:28.978565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:28.978532    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:30.561963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:30.561931    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:31.647034 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:31.647006    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:31.659942 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:31.659919    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:31.958444 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:31.958364    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:32.103229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.102816    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:32.322029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.321953    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:49:32.381464 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:49:32.378504    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:49:32.615823 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.615727    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"]
Oct 30 05:49:32.621662 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.621637    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.640574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.640537    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"]
Oct 30 05:49:32.715706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.715670    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.716069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.715722    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.716069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.715798    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.716069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.715830    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.716069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.715855    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.716069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.715970    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9bh\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-kube-api-access-vc9bh\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.716069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.715999    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.716320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.716107    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.716320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.716146    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817517 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817487    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817689 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817525    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817689 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817543    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817689 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817562    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817689 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817616    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9bh\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-kube-api-access-vc9bh\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817689 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817641    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817738    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817763    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817835    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817905    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.817968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.817962    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.818259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.818142    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.818259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.818243    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.818509 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.818455    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.820102 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.820075    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.820354 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.820337    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.840680 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.840652    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.840868 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.840847    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9bh\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-kube-api-access-vc9bh\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:32.935001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:32.934968    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:33.261751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:33.261721    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"]
Oct 30 05:49:33.262055 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:49:33.261893    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3a5c58_566a_4db2_a801_33bd0c7daa5e.slice/crio-28e2156cf2f089ea5a66c56325b13f2bc439ea568f0cc7f46910e6cd6f220e7a WatchSource:0}: Error finding container 28e2156cf2f089ea5a66c56325b13f2bc439ea568f0cc7f46910e6cd6f220e7a: Status 404 returned error can't find the container with id 28e2156cf2f089ea5a66c56325b13f2bc439ea568f0cc7f46910e6cd6f220e7a
Oct 30 05:49:33.264909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:33.264880    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:49:33.265002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:33.264945    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:49:33.265002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:33.264973    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:49:33.318268 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:33.318237    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" event={"ID":"4c3a5c58-566a-4db2-a801-33bd0c7daa5e","Type":"ContainerStarted","Data":"28e2156cf2f089ea5a66c56325b13f2bc439ea568f0cc7f46910e6cd6f220e7a"}
Oct 30 05:49:33.658712 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:33.658669    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:34.327032 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:34.326995    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" event={"ID":"4c3a5c58-566a-4db2-a801-33bd0c7daa5e","Type":"ContainerStarted","Data":"bd53117abd727563e62cb6e59bea3311b0796cd9390deda33c9fea337987640a"}
Oct 30 05:49:34.420046 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:34.419996    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" podStartSLOduration=2.419982253 podStartE2EDuration="2.419982253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:34.418854982 +0000 UTC m=+1089.249892198" watchObservedRunningTime="2026-10-30 05:49:34.419982253 +0000 UTC m=+1089.251019469"
Oct 30 05:49:34.935558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:34.935515    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:34.937114 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:34.937093    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:49:34.937198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:34.937151    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:49:35.626252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:35.626212    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:35.936123 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:35.936088    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:49:35.938287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:35.936139    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:49:36.936344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:36.936309    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:49:36.936830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:36.936371    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:49:37.171600 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:37.171562    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:37.935341 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:37.935305    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:49:37.935553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:37.935358    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:49:38.899326 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:38.899294    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:38.935788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:38.935746    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:49:38.935958 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:38.935807    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:49:39.936024 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:39.935993    5250 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:49:39.936455 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:39.936052    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:49:40.479449 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:40.479412    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:40.939743 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:40.939713    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:40.940222 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:40.940172    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:40.940945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:40.940929    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:49:42.382842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:42.382804    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:43.493793 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.493755    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"]
Oct 30 05:49:43.513207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.511465    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.540330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.540290    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"]
Oct 30 05:49:43.542741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.542715    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 05:49:43.542945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.542931    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:49:43.597738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.597700    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.615265 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.615211    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.615766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.615740    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mck9d\" (UniqueName: \"kubernetes.io/projected/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-kube-api-access-mck9d\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.767262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.724135    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.767262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.724313    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.767262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.724496    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mck9d\" (UniqueName: \"kubernetes.io/projected/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-kube-api-access-mck9d\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.767262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.755251    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.819410 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.819342    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:43.875111 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:43.875075    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mck9d\" (UniqueName: \"kubernetes.io/projected/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-kube-api-access-mck9d\") pod \"maas-api-revoke-sub-f09b5967-5scmp\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:44.056509 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:44.056411    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:44.732732 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:44.732632    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"]
Oct 30 05:49:44.810784 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:44.809316    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:44.981589 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:44.981266    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:45.144077 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.143301    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:45.145881 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.145262    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:45.280783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.279708    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 05:49:45.351265 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.315291    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:45.389681 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.363302    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:49:45.631257 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.628253    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:45.822418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.822373    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:45.828918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.827754    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:49:45.957201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.947632    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:45.981205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.972643    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp" event={"ID":"8703bc1f-eb3b-4e3d-9a02-f406819a25ea","Type":"ContainerStarted","Data":"2f28fe56494b61a2f64245ac21c85eb217e232d2eed58d5c41c40a9109aa4684"}
Oct 30 05:49:45.981205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:45.975117    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 05:49:46.052677 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.052578    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:49:46.160227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.149292    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:49:46.160227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.150053    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:49:46.160227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.151603    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:46.160227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.151917    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:49:46.285800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.285704    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 05:49:46.286660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.286638    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:49:46.350437 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.350392    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 05:49:46.379038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.378998    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:49:46.379574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.379543    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 05:49:46.379991 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.379966    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:49:46.380722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.380702    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:49:46.390228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.390171    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:49:46.390776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.390754    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:49:46.492082 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.492054    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:49:46.505215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.498748    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:49:46.510506 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.510484    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:49:46.607652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.607621    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:49:46.655768 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.655741    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:49:46.819791 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.819308    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:49:46.981728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.981694    5250 generic.go:356] "Generic (PLEG): container finished" podID="8703bc1f-eb3b-4e3d-9a02-f406819a25ea" containerID="c0b68b32da7be7da10bc9ea066553d60f2d0248432ec19a742981bd34c5c48b7" exitCode=0
Oct 30 05:49:46.982163 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:46.981773    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp" event={"ID":"8703bc1f-eb3b-4e3d-9a02-f406819a25ea","Type":"ContainerDied","Data":"c0b68b32da7be7da10bc9ea066553d60f2d0248432ec19a742981bd34c5c48b7"}
Oct 30 05:49:48.135353 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.135326    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:48.203204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.203161    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-kube-api-access-mck9d\" (UniqueName: \"kubernetes.io/projected/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-kube-api-access-mck9d\") pod \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") "
Oct 30 05:49:48.203382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.203274    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-openshift-ca\") pod \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") "
Oct 30 05:49:48.203382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.203376    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-xks-ca\") pod \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\" (UID: \"8703bc1f-eb3b-4e3d-9a02-f406819a25ea\") "
Oct 30 05:49:48.203684 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.203651    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-openshift-ca" pod "8703bc1f-eb3b-4e3d-9a02-f406819a25ea" (UID: "8703bc1f-eb3b-4e3d-9a02-f406819a25ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:49:48.203955 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.203937    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:49:48.205635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.205610    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-xks-ca" pod "8703bc1f-eb3b-4e3d-9a02-f406819a25ea" (UID: "8703bc1f-eb3b-4e3d-9a02-f406819a25ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:49:48.205729 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.205641    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-kube-api-access-mck9d" pod "8703bc1f-eb3b-4e3d-9a02-f406819a25ea" (UID: "8703bc1f-eb3b-4e3d-9a02-f406819a25ea"). InnerVolumeSpecName "kube-api-access-mck9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:49:48.305802 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.305725    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:49:48.305802 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.305752    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mck9d\" (UniqueName: \"kubernetes.io/projected/8703bc1f-eb3b-4e3d-9a02-f406819a25ea-kube-api-access-mck9d\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:49:48.997635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.997607    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"
Oct 30 05:49:48.997805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.997608    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp" event={"ID":"8703bc1f-eb3b-4e3d-9a02-f406819a25ea","Type":"ContainerDied","Data":"2f28fe56494b61a2f64245ac21c85eb217e232d2eed58d5c41c40a9109aa4684"}
Oct 30 05:49:48.997805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:48.997730    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f28fe56494b61a2f64245ac21c85eb217e232d2eed58d5c41c40a9109aa4684"
Oct 30 05:49:49.079301 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:49.079274    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:50.469367 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:50.469327    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:52.082438 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:52.082395    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:53.657216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:53.657156    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:55.350748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:55.350716    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:56.962759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:56.962727    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:58.557501 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:49:58.557471    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:00.172422 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:00.172391    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:01.881877 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:01.881844    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:02.375507 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:50:02.375473    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:50:03.038033 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.038002    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"]
Oct 30 05:50:03.038695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.038678    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8703bc1f-eb3b-4e3d-9a02-f406819a25ea" containerName="revoke-keys"
Oct 30 05:50:03.038695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.038695    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8703bc1f-eb3b-4e3d-9a02-f406819a25ea" containerName="revoke-keys"
Oct 30 05:50:03.038844 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.038808    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8703bc1f-eb3b-4e3d-9a02-f406819a25ea" containerName="revoke-keys"
Oct 30 05:50:03.043754 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.043736    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.069217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.069155    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"]
Oct 30 05:50:03.091762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.091733    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.092087 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.092058    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.092342 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.092316    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.092417 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.092381    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.092473 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.092423    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.092526 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.092509    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.092584 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.092571    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.092636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.092595    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g687b\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-kube-api-access-g687b\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.092704 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.092651    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.193910 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.193876    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.193944    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.193983    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.193998    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g687b\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-kube-api-access-g687b\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194033    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194171    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194242    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194410    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194409    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194484    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194717 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194696    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194719    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194847 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194820    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.194904 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.194880    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.196661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.196631    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.197069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.197052    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.217211 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.217153    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.217339 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.217252    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g687b\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-kube-api-access-g687b\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.321031 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.320927    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:03.360219 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.360168    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:03.722836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.722805    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"]
Oct 30 05:50:03.722989 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:50:03.722963    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4eba3a_3cac_4cc6_8fbb_b114f1eeb931.slice/crio-88eedb0a8e0232fed7aa66175209654f45503e5809ee271ca22b65071ac8465b WatchSource:0}: Error finding container 88eedb0a8e0232fed7aa66175209654f45503e5809ee271ca22b65071ac8465b: Status 404 returned error can't find the container with id 88eedb0a8e0232fed7aa66175209654f45503e5809ee271ca22b65071ac8465b
Oct 30 05:50:03.725687 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.725651    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:03.725786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.725731    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:03.725786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:03.725774    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:04.103410 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.103324    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" event={"ID":"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931","Type":"ContainerStarted","Data":"14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0"}
Oct 30 05:50:04.103410 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.103390    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" event={"ID":"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931","Type":"ContainerStarted","Data":"88eedb0a8e0232fed7aa66175209654f45503e5809ee271ca22b65071ac8465b"}
Oct 30 05:50:04.161396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.161343    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podStartSLOduration=1.161327952 podStartE2EDuration="1.161327952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:04.158931103 +0000 UTC m=+1118.989968321" watchObservedRunningTime="2026-10-30 05:50:04.161327952 +0000 UTC m=+1118.992365167"
Oct 30 05:50:04.281981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.281951    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"]
Oct 30 05:50:04.286370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.286345    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.303293 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.303246    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"distinct-2-w2-e7a506-kserve-self-signed-certs\""
Oct 30 05:50:04.308158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.308130    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"]
Oct 30 05:50:04.309903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.309872    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-home\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.310084 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.309957    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7sg\" (UniqueName: \"kubernetes.io/projected/c8824800-8838-4200-8f50-4927e8075d3b-kube-api-access-xf7sg\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.310084 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.310062    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-model-cache\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.310325 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.310118    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8824800-8838-4200-8f50-4927e8075d3b-tls-certs\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.310325 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.310152    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-dshm\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.310325 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.310198    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-tmp-dir\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.361241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.361132    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:04.362867 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.362838    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:50:04.362999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.362890    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:50:04.390714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.390668    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:04.418204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.414363    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7sg\" (UniqueName: \"kubernetes.io/projected/c8824800-8838-4200-8f50-4927e8075d3b-kube-api-access-xf7sg\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.418204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.414877    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-model-cache\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.418204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.414958    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8824800-8838-4200-8f50-4927e8075d3b-tls-certs\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.418204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.415004    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-dshm\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.418204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.415033    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-tmp-dir\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.418204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.415284    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-home\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.418204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.415960    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-home\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.418204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.416332    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-model-cache\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.418977 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.418952    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-tmp-dir\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.421320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.421298    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8824800-8838-4200-8f50-4927e8075d3b-tls-certs\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.424092 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.424074    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-dshm\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.440342 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.440286    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7sg\" (UniqueName: \"kubernetes.io/projected/c8824800-8838-4200-8f50-4927e8075d3b-kube-api-access-xf7sg\") pod \"distinct-2-w2-e7a506-kserve-595954758d-lvgmk\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") " pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:04.606902 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:04.604562    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:05.052048 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:05.052018    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"]
Oct 30 05:50:05.052674 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:50:05.052649    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8824800_8838_4200_8f50_4927e8075d3b.slice/crio-13e8f5ba25ec3077f2c318b80011b3427c323a3f436cdb7a5992dc7be8dbdddd WatchSource:0}: Error finding container 13e8f5ba25ec3077f2c318b80011b3427c323a3f436cdb7a5992dc7be8dbdddd: Status 404 returned error can't find the container with id 13e8f5ba25ec3077f2c318b80011b3427c323a3f436cdb7a5992dc7be8dbdddd
Oct 30 05:50:05.112118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:05.112089    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" event={"ID":"c8824800-8838-4200-8f50-4927e8075d3b","Type":"ContainerStarted","Data":"13e8f5ba25ec3077f2c318b80011b3427c323a3f436cdb7a5992dc7be8dbdddd"}
Oct 30 05:50:05.181722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:05.181687    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:05.346091 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:05.340576    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:05.364720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:05.364677    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:50:05.365020 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:05.364988    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:50:05.896360 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:05.896329    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:06.120064 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:06.120027    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" event={"ID":"c8824800-8838-4200-8f50-4927e8075d3b","Type":"ContainerStarted","Data":"e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f"}
Oct 30 05:50:06.362933 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:06.362841    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:50:06.363210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:06.363156    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:50:06.605462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:06.605424    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:07.208427 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.208396    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:07.365202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.360564    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:50:07.365202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.360625    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:50:07.514043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.513948    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podStartSLOduration=3.513933363 podStartE2EDuration="3.513933363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:06.180600525 +0000 UTC m=+1121.011637755" watchObservedRunningTime="2026-10-30 05:50:07.513933363 +0000 UTC m=+1122.344970579"
Oct 30 05:50:07.515307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.515283    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"]
Oct 30 05:50:07.519427 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.519412    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.539409 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.539381    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"]
Oct 30 05:50:07.546008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.545983    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:50:07.546215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.546196    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 05:50:07.597161 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.597119    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.597161 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.597156    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgtxg\" (UniqueName: \"kubernetes.io/projected/3c82d544-c33b-4567-a811-4e21a7bf975a-kube-api-access-jgtxg\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.597455 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.597266    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.698457 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.698424    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.698457 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.698462    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgtxg\" (UniqueName: \"kubernetes.io/projected/3c82d544-c33b-4567-a811-4e21a7bf975a-kube-api-access-jgtxg\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.698715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.698500    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.699508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.699475    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.701004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.700985    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.722109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.722074    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgtxg\" (UniqueName: \"kubernetes.io/projected/3c82d544-c33b-4567-a811-4e21a7bf975a-kube-api-access-jgtxg\") pod \"maas-api-revoke-sub-736a78db-bprfv\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:07.835500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:07.835422    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:08.183738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:08.183706    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"]
Oct 30 05:50:08.185664 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:50:08.185639    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c82d544_c33b_4567_a811_4e21a7bf975a.slice/crio-183ea3ee9e045403b3e5bb440e7ebf3ab95cdd4eb48fc40f6a7333d16fb9f37d WatchSource:0}: Error finding container 183ea3ee9e045403b3e5bb440e7ebf3ab95cdd4eb48fc40f6a7333d16fb9f37d: Status 404 returned error can't find the container with id 183ea3ee9e045403b3e5bb440e7ebf3ab95cdd4eb48fc40f6a7333d16fb9f37d
Oct 30 05:50:08.378205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:08.371949    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:50:08.378205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:08.372007    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:50:08.983354 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:08.983321    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:09.141390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:09.141349    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv" event={"ID":"3c82d544-c33b-4567-a811-4e21a7bf975a","Type":"ContainerStarted","Data":"aaf623bc9e515b97961c9c1266d7ed683ec0f9b151839d3893a80cf7e034abed"}
Oct 30 05:50:09.141595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:09.141426    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv" event={"ID":"3c82d544-c33b-4567-a811-4e21a7bf975a","Type":"ContainerStarted","Data":"183ea3ee9e045403b3e5bb440e7ebf3ab95cdd4eb48fc40f6a7333d16fb9f37d"}
Oct 30 05:50:09.376202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:09.363547    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:50:09.376202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:09.363621    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:50:10.150913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:10.150879    5250 generic.go:356] "Generic (PLEG): container finished" podID="3c82d544-c33b-4567-a811-4e21a7bf975a" containerID="aaf623bc9e515b97961c9c1266d7ed683ec0f9b151839d3893a80cf7e034abed" exitCode=0
Oct 30 05:50:10.151292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:10.150916    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv" event={"ID":"3c82d544-c33b-4567-a811-4e21a7bf975a","Type":"ContainerDied","Data":"aaf623bc9e515b97961c9c1266d7ed683ec0f9b151839d3893a80cf7e034abed"}
Oct 30 05:50:10.360540 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:10.360506    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:50:10.360706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:10.360579    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:50:10.693718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:10.693676    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:11.383425 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.381502    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:11.391974 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.391949    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:11.406012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.404445    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:50:11.699230 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.696352    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:11.819950 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.819911    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-xks-ca\") pod \"3c82d544-c33b-4567-a811-4e21a7bf975a\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") "
Oct 30 05:50:11.820695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.820667    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-openshift-ca\") pod \"3c82d544-c33b-4567-a811-4e21a7bf975a\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") "
Oct 30 05:50:11.846253 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.821049    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c82d544-c33b-4567-a811-4e21a7bf975a-kube-api-access-jgtxg\" (UniqueName: \"kubernetes.io/projected/3c82d544-c33b-4567-a811-4e21a7bf975a-kube-api-access-jgtxg\") pod \"3c82d544-c33b-4567-a811-4e21a7bf975a\" (UID: \"3c82d544-c33b-4567-a811-4e21a7bf975a\") "
Oct 30 05:50:11.903427 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.889404    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-openshift-ca" pod "3c82d544-c33b-4567-a811-4e21a7bf975a" (UID: "3c82d544-c33b-4567-a811-4e21a7bf975a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:50:11.971204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.940250    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c82d544-c33b-4567-a811-4e21a7bf975a-kube-api-access-jgtxg" pod "3c82d544-c33b-4567-a811-4e21a7bf975a" (UID: "3c82d544-c33b-4567-a811-4e21a7bf975a"). InnerVolumeSpecName "kube-api-access-jgtxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:50:12.002206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.980566    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgtxg\" (UniqueName: \"kubernetes.io/projected/3c82d544-c33b-4567-a811-4e21a7bf975a-kube-api-access-jgtxg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:50:12.002206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:11.980601    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:50:12.032630 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.032402    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-xks-ca" pod "3c82d544-c33b-4567-a811-4e21a7bf975a" (UID: "3c82d544-c33b-4567-a811-4e21a7bf975a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:50:12.089204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.086392    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c82d544-c33b-4567-a811-4e21a7bf975a-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:50:12.594211 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.593423    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:12.708199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.707652    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:12.720204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.713205    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 30 05:50:12.739786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.737934    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:12.837240 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.834406    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:12.879259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.870753    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"
Oct 30 05:50:12.879259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.873456    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv" event={"ID":"3c82d544-c33b-4567-a811-4e21a7bf975a","Type":"ContainerDied","Data":"183ea3ee9e045403b3e5bb440e7ebf3ab95cdd4eb48fc40f6a7333d16fb9f37d"}
Oct 30 05:50:12.879259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.873523    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="183ea3ee9e045403b3e5bb440e7ebf3ab95cdd4eb48fc40f6a7333d16fb9f37d"
Oct 30 05:50:12.914209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.914070    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:12.998690 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:12.997528    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.060311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.060256    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.308236 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.305353    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-drcjl" podUID="a55f3b2c-a554-448d-8455-d4bd9c51aff9" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.14:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.520478 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.518284    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.564286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.549153    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.589361 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.589311    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.597218 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.595295    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.597218 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.595609    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.597218 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.595875    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.597218 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.596201    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:50:13.637011 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.636037    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 05:50:13.678443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.678307    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:50:13.678443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.678365    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.853074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.852971    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 05:50:13.865204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.854706    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 05:50:13.865719 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.861209    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:13.866149 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.866129    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:50:13.866334 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:13.866321    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:50:14.038201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.023637    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:14.038201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.027138    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:50:14.083594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.083546    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:14.094254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.094219    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:50:14.397449 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.371358    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:50:14.574204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.573263    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:14.576817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.574682    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:50:14.606216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.604509    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:14.606714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.606538    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:50:14.614210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.608017    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:50:14.614210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.609898    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:14.618202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.615746    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:50:14.618202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.615784    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:14.684563 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.683001    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:14.684563 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.684170    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:50:14.884201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.869107    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:14.977259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:14.976837    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:50:15.024202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.024030    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Oct 30 05:50:15.043204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.024679    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:15.128204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.124239    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:50:15.128204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.124311    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:15.128204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.124611    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:15.397336 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.344208    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:15.428449 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.428379    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:15.524874 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.524832    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:50:15.589205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.582924    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:15.616527 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.616482    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 05:50:15.630204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.619042    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:50:15.675464 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.675335    5250 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:15.722304 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.722258    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:15.886328 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.886270    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:50:15.999531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:15.999433    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:16.028367 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.028322    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:16.135476 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.135410    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:16.372261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.350872    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:16.372261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.353707    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:50:16.508103 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.508061    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:16.508460 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.508428    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:16.509165 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.509143    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:50:16.552381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.532359    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:50:16.552381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.533148    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:50:16.580571 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.580531    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:16.615539 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.590697    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 05:50:16.617411 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.617376    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:16.646691 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.635901    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:16.758204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:16.754274    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 05:50:17.087264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.072648    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:50:17.087264 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.084860    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:17.089293 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.089257    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:17.091253 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.091072    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:50:17.195546 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.195508    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:50:17.536605 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.536566    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:17.564940 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.563972    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:17.617776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.617736    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:17.646702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.646659    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:17.711198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.701912    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 05:50:17.744482 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.744447    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 05:50:17.745357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.745333    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 05:50:17.745535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.745524    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 05:50:17.747246 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.747228    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:50:17.747390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.747378    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 05:50:17.821731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.821592    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:50:17.850361 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.850327    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:50:17.903779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.903751    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:50:17.904631 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.904600    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:50:17.929813 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.929784    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 05:50:17.932069 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:17.932045    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:50:18.387952 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:18.387922    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:18.565709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:18.565674    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:20.886377 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:20.886338    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:21.042209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:21.042143    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:21.179288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:21.179200    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:21.348667 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:21.348581    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:21.536447 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:21.536362    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:21.711106 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:21.711074    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:21.864836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:21.864710    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:22.415930 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:22.415895    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:22.568788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:22.568751    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:22.710434 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:22.710300    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:23.607252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:23.607213    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:24.093695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.093665    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8874-acb3cb"]
Oct 30 05:50:24.094577 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.094550    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3c82d544-c33b-4567-a811-4e21a7bf975a" containerName="revoke-keys"
Oct 30 05:50:24.094577 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.094577    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c82d544-c33b-4567-a811-4e21a7bf975a" containerName="revoke-keys"
Oct 30 05:50:24.094831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.094813    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c82d544-c33b-4567-a811-4e21a7bf975a" containerName="revoke-keys"
Oct 30 05:50:24.099011 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.098992    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-acb3cb"
Oct 30 05:50:24.117848 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.117822    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-acb3cb"]
Oct 30 05:50:24.122804 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.122783    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bnq\""
Oct 30 05:50:24.157529 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.157498    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46hm\" (UniqueName: \"kubernetes.io/projected/310e0812-1eb8-40a2-9107-635f225e8c17-kube-api-access-k46hm\") pod \"test-curl-8874-acb3cb\" (UID: \"310e0812-1eb8-40a2-9107-635f225e8c17\") " pod="opendatahub/test-curl-8874-acb3cb"
Oct 30 05:50:24.195868 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.195834    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:24.259279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.259243    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46hm\" (UniqueName: \"kubernetes.io/projected/310e0812-1eb8-40a2-9107-635f225e8c17-kube-api-access-k46hm\") pod \"test-curl-8874-acb3cb\" (UID: \"310e0812-1eb8-40a2-9107-635f225e8c17\") " pod="opendatahub/test-curl-8874-acb3cb"
Oct 30 05:50:24.285304 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.285276    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46hm\" (UniqueName: \"kubernetes.io/projected/310e0812-1eb8-40a2-9107-635f225e8c17-kube-api-access-k46hm\") pod \"test-curl-8874-acb3cb\" (UID: \"310e0812-1eb8-40a2-9107-635f225e8c17\") " pod="opendatahub/test-curl-8874-acb3cb"
Oct 30 05:50:24.364420 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.364324    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:24.414662 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.414626    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-acb3cb"
Oct 30 05:50:24.691737 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.690700    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:24.709432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.709403    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:50:24.894978 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.894887    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-acb3cb"]
Oct 30 05:50:24.896071 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:50:24.896040    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod310e0812_1eb8_40a2_9107_635f225e8c17.slice/crio-e012d590dfaf8bd6b09265e634b9d63e591c0ffe8392a38d06bcb30478acc746 WatchSource:0}: Error finding container e012d590dfaf8bd6b09265e634b9d63e591c0ffe8392a38d06bcb30478acc746: Status 404 returned error can't find the container with id e012d590dfaf8bd6b09265e634b9d63e591c0ffe8392a38d06bcb30478acc746
Oct 30 05:50:24.898213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:24.898157    5250 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:50:25.151773 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:25.151697    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:25.296784 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:25.296755    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:25.704107 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:25.704016    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-acb3cb" event={"ID":"310e0812-1eb8-40a2-9107-635f225e8c17","Type":"ContainerStarted","Data":"ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744"}
Oct 30 05:50:25.704107 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:25.704057    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-acb3cb" event={"ID":"310e0812-1eb8-40a2-9107-635f225e8c17","Type":"ContainerStarted","Data":"e012d590dfaf8bd6b09265e634b9d63e591c0ffe8392a38d06bcb30478acc746"}
Oct 30 05:50:25.742121 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:25.742067    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8874-acb3cb" podStartSLOduration=1.2812875959999999 podStartE2EDuration="1.742053128s" totalImagesPullingTime="460.765532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:25.740123823 +0000 UTC m=+1140.571161041" watchObservedRunningTime="2026-10-30 05:50:25.742053128 +0000 UTC m=+1140.573090345"
Oct 30 05:50:25.750440 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:25.750411    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:26.115969 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.115902    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:26.258611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.258581    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"]
Oct 30 05:50:26.262749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.262723    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:26.262876 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.262862    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:26.281012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.280987    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3cc280cf\""
Oct 30 05:50:26.294220 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.292567    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3cc280cf-dockercfg-j8x4z\""
Oct 30 05:50:26.296798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.296772    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"]
Oct 30 05:50:26.310052 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.310027    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b0bac3-2617-4505-b1ab-6f49cbd83244-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp\" (UID: \"42b0bac3-2617-4505-b1ab-6f49cbd83244\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:26.310168 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.310108    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnkz6\" (UniqueName: \"kubernetes.io/projected/42b0bac3-2617-4505-b1ab-6f49cbd83244-kube-api-access-dnkz6\") pod \"payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp\" (UID: \"42b0bac3-2617-4505-b1ab-6f49cbd83244\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:26.400804 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.397658    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"]
Oct 30 05:50:26.410933 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.410081    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:26.411666 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.411637    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnkz6\" (UniqueName: \"kubernetes.io/projected/42b0bac3-2617-4505-b1ab-6f49cbd83244-kube-api-access-dnkz6\") pod \"payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp\" (UID: \"42b0bac3-2617-4505-b1ab-6f49cbd83244\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:26.412076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.411895    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b0bac3-2617-4505-b1ab-6f49cbd83244-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp\" (UID: \"42b0bac3-2617-4505-b1ab-6f49cbd83244\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:26.424936 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.423665    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b0bac3-2617-4505-b1ab-6f49cbd83244-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp\" (UID: \"42b0bac3-2617-4505-b1ab-6f49cbd83244\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:26.459555 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.445623    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnkz6\" (UniqueName: \"kubernetes.io/projected/42b0bac3-2617-4505-b1ab-6f49cbd83244-kube-api-access-dnkz6\") pod \"payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp\" (UID: \"42b0bac3-2617-4505-b1ab-6f49cbd83244\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:26.459555 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.451114    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"]
Oct 30 05:50:26.513563 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.513526    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzbrv\" (UniqueName: \"kubernetes.io/projected/504727f8-a865-4093-88db-840e38f8fad7-kube-api-access-rzbrv\") pod \"payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp\" (UID: \"504727f8-a865-4093-88db-840e38f8fad7\") " pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:26.513773 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.513724    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504727f8-a865-4093-88db-840e38f8fad7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp\" (UID: \"504727f8-a865-4093-88db-840e38f8fad7\") " pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:26.576945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.576914    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:26.615440 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.615387    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzbrv\" (UniqueName: \"kubernetes.io/projected/504727f8-a865-4093-88db-840e38f8fad7-kube-api-access-rzbrv\") pod \"payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp\" (UID: \"504727f8-a865-4093-88db-840e38f8fad7\") " pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:26.615610 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.615530    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504727f8-a865-4093-88db-840e38f8fad7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp\" (UID: \"504727f8-a865-4093-88db-840e38f8fad7\") " pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:26.616235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.616210    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504727f8-a865-4093-88db-840e38f8fad7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp\" (UID: \"504727f8-a865-4093-88db-840e38f8fad7\") " pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:26.670925 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.670886    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzbrv\" (UniqueName: \"kubernetes.io/projected/504727f8-a865-4093-88db-840e38f8fad7-kube-api-access-rzbrv\") pod \"payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp\" (UID: \"504727f8-a865-4093-88db-840e38f8fad7\") " pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:26.757319 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.757286    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:26.772813 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.772787    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8874-acb3cb"]
Oct 30 05:50:26.783725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.783693    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8874-acb3cb"]
Oct 30 05:50:26.891801 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:26.891770    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"]
Oct 30 05:50:26.892277 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:50:26.892247    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b0bac3_2617_4505_b1ab_6f49cbd83244.slice/crio-689ce4a431619ee3ff43f77c23c312c01bbaf0f9577c8a6b28909739dce63cbf WatchSource:0}: Error finding container 689ce4a431619ee3ff43f77c23c312c01bbaf0f9577c8a6b28909739dce63cbf: Status 404 returned error can't find the container with id 689ce4a431619ee3ff43f77c23c312c01bbaf0f9577c8a6b28909739dce63cbf
Oct 30 05:50:27.076657 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.076629    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"]
Oct 30 05:50:27.077358 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:50:27.077326    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504727f8_a865_4093_88db_840e38f8fad7.slice/crio-4afc94b721e5a102bda20251c0ad62dab0d688b8b682db3d6c8526ab41fa5c80 WatchSource:0}: Error finding container 4afc94b721e5a102bda20251c0ad62dab0d688b8b682db3d6c8526ab41fa5c80: Status 404 returned error can't find the container with id 4afc94b721e5a102bda20251c0ad62dab0d688b8b682db3d6c8526ab41fa5c80
Oct 30 05:50:27.500026 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.499989    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:27.662199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.657851    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:27.722428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.722396    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp" event={"ID":"504727f8-a865-4093-88db-840e38f8fad7","Type":"ContainerStarted","Data":"d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1"}
Oct 30 05:50:27.722634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.722456    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp" event={"ID":"504727f8-a865-4093-88db-840e38f8fad7","Type":"ContainerStarted","Data":"4afc94b721e5a102bda20251c0ad62dab0d688b8b682db3d6c8526ab41fa5c80"}
Oct 30 05:50:27.722634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.722607    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:27.724654 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.724624    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp" event={"ID":"42b0bac3-2617-4505-b1ab-6f49cbd83244","Type":"ContainerStarted","Data":"437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8"}
Oct 30 05:50:27.724819 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.724680    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp" event={"ID":"42b0bac3-2617-4505-b1ab-6f49cbd83244","Type":"ContainerStarted","Data":"689ce4a431619ee3ff43f77c23c312c01bbaf0f9577c8a6b28909739dce63cbf"}
Oct 30 05:50:27.724819 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.724769    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:27.724946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.724868    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8874-acb3cb" podUID="310e0812-1eb8-40a2-9107-635f225e8c17" containerName="test-curl-8874-acb3cb" containerID="cri-o://ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744" gracePeriod=2
Oct 30 05:50:27.766734 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.766625    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp" podStartSLOduration=1.766610322 podStartE2EDuration="1.766610322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:27.765116738 +0000 UTC m=+1142.596153956" watchObservedRunningTime="2026-10-30 05:50:27.766610322 +0000 UTC m=+1142.597647539"
Oct 30 05:50:27.807400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:27.807343    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp" podStartSLOduration=1.8073277189999999 podStartE2EDuration="1.807327719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:27.805151405 +0000 UTC m=+1142.636188626" watchObservedRunningTime="2026-10-30 05:50:27.807327719 +0000 UTC m=+1142.638364935"
Oct 30 05:50:29.177121 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:29.177084    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:29.931170 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:29.931136    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-acb3cb"
Oct 30 05:50:30.064761 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.064681    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310e0812-1eb8-40a2-9107-635f225e8c17-kube-api-access-k46hm\" (UniqueName: \"kubernetes.io/projected/310e0812-1eb8-40a2-9107-635f225e8c17-kube-api-access-k46hm\") pod \"310e0812-1eb8-40a2-9107-635f225e8c17\" (UID: \"310e0812-1eb8-40a2-9107-635f225e8c17\") "
Oct 30 05:50:30.066902 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.066875    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310e0812-1eb8-40a2-9107-635f225e8c17-kube-api-access-k46hm" pod "310e0812-1eb8-40a2-9107-635f225e8c17" (UID: "310e0812-1eb8-40a2-9107-635f225e8c17"). InnerVolumeSpecName "kube-api-access-k46hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:50:30.166206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.166150    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k46hm\" (UniqueName: \"kubernetes.io/projected/310e0812-1eb8-40a2-9107-635f225e8c17-kube-api-access-k46hm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:50:30.752532 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.752501    5250 generic.go:356] "Generic (PLEG): container finished" podID="310e0812-1eb8-40a2-9107-635f225e8c17" containerID="ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744" exitCode=137
Oct 30 05:50:30.753009 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.752549    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-acb3cb"
Oct 30 05:50:30.753009 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.752603    5250 scope.go:122] "RemoveContainer" containerID="ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744"
Oct 30 05:50:30.763495 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.763481    5250 scope.go:122] "RemoveContainer" containerID="ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744"
Oct 30 05:50:30.763767 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:50:30.763747    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744\": container with ID starting with ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744 not found: ID does not exist" containerID="ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744"
Oct 30 05:50:30.763827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.763775    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744"} err="failed to get container status \"ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744\": rpc error: code = NotFound desc = could not find container \"ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744\": container with ID starting with ece7a7d9193371215ee5cff2030b1e9486ca95e7f8916cbcbb52caa4a376a744 not found: ID does not exist"
Oct 30 05:50:30.773866 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:50:30.773841    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-acb3cb\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="310e0812-1eb8-40a2-9107-635f225e8c17" pod="opendatahub/test-curl-8874-acb3cb"
Oct 30 05:50:30.774875 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.774858    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:30.934476 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:30.934446    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:31.281601 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:31.281568    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310e0812-1eb8-40a2-9107-635f225e8c17" path="/var/lib/kubelet/pods/310e0812-1eb8-40a2-9107-635f225e8c17/volumes"
Oct 30 05:50:32.068914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:32.068881    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:32.373740 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:50:32.373664    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:50:32.435976 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:32.435946    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:32.911462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:32.911428    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"]
Oct 30 05:50:32.912094 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:32.912079    5250 cpu_manager.go:414] "containerMap: removing container" podUID="310e0812-1eb8-40a2-9107-635f225e8c17" containerName="test-curl-8874-acb3cb"
Oct 30 05:50:32.912137 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:32.912097    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310e0812-1eb8-40a2-9107-635f225e8c17" containerName="test-curl-8874-acb3cb"
Oct 30 05:50:32.912248 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:32.912234    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="310e0812-1eb8-40a2-9107-635f225e8c17" containerName="test-curl-8874-acb3cb"
Oct 30 05:50:32.917097 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:32.917082    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:32.954632 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:32.954602    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"]
Oct 30 05:50:33.005506 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005476    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.005670 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005523    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.005670 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005546    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.005802 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005683    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.005852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005798    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.005905 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005872    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.005954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005918    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2zf\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-kube-api-access-ln2zf\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.005954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005937    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.006044 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.005954    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d887fd9-40b4-44c7-8c81-89a824d26ace-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.107738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.107706    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.107747    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2zf\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-kube-api-access-ln2zf\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.107771    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.107790    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d887fd9-40b4-44c7-8c81-89a824d26ace-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.107905    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.107958    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.107983    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.108043    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108232 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.108169    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.108287    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.108341    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.108485    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.108596    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.108715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.108696    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d887fd9-40b4-44c7-8c81-89a824d26ace-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.110690 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.110661    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.110922 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.110898    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.130571 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.130550    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.131009 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.130985    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2zf\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-kube-api-access-ln2zf\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.231916 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.231833    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:33.554378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.554345    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"]
Oct 30 05:50:33.555015 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:50:33.554988    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d887fd9_40b4_44c7_8c81_89a824d26ace.slice/crio-a5b21b8e28e46d749afbb4070373f84cb09910353553350f17ffa3f2c7485899 WatchSource:0}: Error finding container a5b21b8e28e46d749afbb4070373f84cb09910353553350f17ffa3f2c7485899: Status 404 returned error can't find the container with id a5b21b8e28e46d749afbb4070373f84cb09910353553350f17ffa3f2c7485899
Oct 30 05:50:33.557679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.557647    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:33.557788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.557716    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:33.557788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.557745    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:33.558015 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.557990    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:33.700469 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.700439    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:33.776810 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.776768    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" event={"ID":"8d887fd9-40b4-44c7-8c81-89a824d26ace","Type":"ContainerStarted","Data":"147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad"}
Oct 30 05:50:33.777016 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.776857    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" event={"ID":"8d887fd9-40b4-44c7-8c81-89a824d26ace","Type":"ContainerStarted","Data":"a5b21b8e28e46d749afbb4070373f84cb09910353553350f17ffa3f2c7485899"}
Oct 30 05:50:33.826233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:33.826121    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podStartSLOduration=1.826106084 podStartE2EDuration="1.826106084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:33.821507884 +0000 UTC m=+1148.652545140" watchObservedRunningTime="2026-10-30 05:50:33.826106084 +0000 UTC m=+1148.657143394"
Oct 30 05:50:34.022502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:34.022461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:34.232301 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:34.232257    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:34.233807 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:34.233776    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 05:50:34.233941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:34.233828    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 05:50:35.232759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:35.232727    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 05:50:35.233153 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:35.232780    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 05:50:35.822709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:35.822672    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:35.989320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:35.989256    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:36.214635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:36.214606    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:36.233039 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:36.233008    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 05:50:36.233396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:36.233058    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 05:50:37.233087 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:37.233054    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 05:50:37.233495 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:37.233119    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 05:50:38.233027 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:38.232985    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 05:50:38.233217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:38.233042    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 05:50:38.502399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:38.502298    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:38.668004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:38.667930    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:38.740454 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:38.740428    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:50:38.740624 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:38.740493    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:50:39.232714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:39.232683    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 05:50:39.232894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:39.232759    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 05:50:40.232503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:40.232471    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 05:50:40.232919 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:40.232523    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 05:50:40.551324 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:40.551236    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:40.700451 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:40.700402    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:41.175764 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:41.175727    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:41.237572 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:41.237546    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:41.237975 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:41.237855    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:41.238546 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:41.238529    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 05:50:42.851389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:42.851336    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:43.677494 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:43.677454    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:44.071300 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.071219    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"]
Oct 30 05:50:44.099429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.099394    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.107829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.107793    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"]
Oct 30 05:50:44.114350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.114322    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-19e33488-openshift-default-dockercfg-s6d7p\""
Oct 30 05:50:44.149636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.149592    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:44.224842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.224801    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab10583-5b3b-4092-83e0-beee39ba10a7-istiod-ca-cert\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.225261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.225235    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-socket\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.225490 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.225472    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbjs\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-kube-api-access-xgbjs\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.225613 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.225597    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-credential-socket\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.225759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.225742    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-data\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.225873 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.225859    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-podinfo\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.226038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.226023    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-envoy\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.226203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.226166    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-certs\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.226357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.226342    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-token\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.307809 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.307777    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:44.328147 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328051    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-socket\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.328147 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328141    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbjs\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-kube-api-access-xgbjs\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.328450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328212    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-credential-socket\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.328450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328274    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-data\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.328450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328316    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-podinfo\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.328450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328388    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-envoy\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.328450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328445    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-certs\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.328699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328506    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-token\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.328699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.328598    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab10583-5b3b-4092-83e0-beee39ba10a7-istiod-ca-cert\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.329666 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.329617    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-socket\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.329788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.329661    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-certs\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.329788 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.329735    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab10583-5b3b-4092-83e0-beee39ba10a7-istiod-ca-cert\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.329895 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.329785    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-credential-socket\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.329968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.329944    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-data\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.332390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.332363    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-envoy\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.333036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.333008    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-podinfo\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.355212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.352419    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-token\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.355212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.352553    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbjs\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-kube-api-access-xgbjs\") pod \"e2e-ait-19e33488-openshift-default-6d958965c5-r5t65\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") " pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.420008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.419965    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:44.607270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.607134    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:44.758567 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.758535    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:44.761154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.761125    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"]
Oct 30 05:50:44.762061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.762033    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:44.762156 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.762107    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:44.762156 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.762136    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:50:44.878818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.878724    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" event={"ID":"6ab10583-5b3b-4092-83e0-beee39ba10a7","Type":"ContainerStarted","Data":"5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a"}
Oct 30 05:50:44.878818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.878798    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" event={"ID":"6ab10583-5b3b-4092-83e0-beee39ba10a7","Type":"ContainerStarted","Data":"b648d33edf24467de3c621f29d2aa2143fc9f603e2c9da0b83af8d3cd39aaa5d"}
Oct 30 05:50:44.929001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:44.928922    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" podStartSLOduration=0.928897378 podStartE2EDuration="928.897378ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:44.924776003 +0000 UTC m=+1159.755813219" watchObservedRunningTime="2026-10-30 05:50:44.928897378 +0000 UTC m=+1159.759934597"
Oct 30 05:50:45.421101 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:45.421066    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:45.426560 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:45.426535    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:45.889684 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:45.889588    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:45.890828 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:45.890807    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:50:46.676559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:46.674563    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:46.943591 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:46.943484    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:47.312263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:47.310002    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:48.936408 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:48.936376    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:49.082704 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:49.082671    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:50.679210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:50.679157    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:51.040710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:51.040600    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:52.102999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:52.102964    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:53.593798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:53.593681    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:55.072224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:55.072137    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:55.354357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:55.354286    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:56.895846 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:56.895802    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:57.700393 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:57.700357    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:58.403537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:50:58.403504    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:00.247229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:00.247153    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:01.768911 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:01.768873    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:02.380621 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:51:02.379497    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:51:03.678359 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:03.678324    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:05.445124 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.445085    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:05.486234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.486203    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"]
Oct 30 05:51:05.498057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.498028    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.515501 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.515460    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"]
Oct 30 05:51:05.525488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.525454    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:51:05.525643 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.525498    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 05:51:05.594126 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.594092    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.594355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.594144    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjtgd\" (UniqueName: \"kubernetes.io/projected/19e9dae2-fad2-4c83-a068-567aeabac096-kube-api-access-wjtgd\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.594355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.594252    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.598789 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.598740    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:05.696448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.696349    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.696448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.696405    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjtgd\" (UniqueName: \"kubernetes.io/projected/19e9dae2-fad2-4c83-a068-567aeabac096-kube-api-access-wjtgd\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.696699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.696501    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.697401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.697297    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.699068 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.699016    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.731027 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.731003    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjtgd\" (UniqueName: \"kubernetes.io/projected/19e9dae2-fad2-4c83-a068-567aeabac096-kube-api-access-wjtgd\") pod \"maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:05.792882 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.792815    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:05.830092 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:05.830064    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:06.137598 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:06.137569    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"]
Oct 30 05:51:06.139789 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:51:06.139764    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9dae2_fad2_4c83_a068_567aeabac096.slice/crio-89437e0ea72c4608aeb76dc940bfa63ea4d339c9e402c5795ea55d435e27fdbe WatchSource:0}: Error finding container 89437e0ea72c4608aeb76dc940bfa63ea4d339c9e402c5795ea55d435e27fdbe: Status 404 returned error can't find the container with id 89437e0ea72c4608aeb76dc940bfa63ea4d339c9e402c5795ea55d435e27fdbe
Oct 30 05:51:07.072472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:07.072435    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j" event={"ID":"19e9dae2-fad2-4c83-a068-567aeabac096","Type":"ContainerStarted","Data":"357505cfcf2bdb8a8ab9bffece83c2c4a47229e0037689f1c1d9fe80287f1285"}
Oct 30 05:51:07.072881 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:07.072500    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j" event={"ID":"19e9dae2-fad2-4c83-a068-567aeabac096","Type":"ContainerStarted","Data":"89437e0ea72c4608aeb76dc940bfa63ea4d339c9e402c5795ea55d435e27fdbe"}
Oct 30 05:51:07.116404 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:07.116354    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j" podStartSLOduration=2.116339712 podStartE2EDuration="2.116339712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:07.114998216 +0000 UTC m=+1181.946035434" watchObservedRunningTime="2026-10-30 05:51:07.116339712 +0000 UTC m=+1181.947376928"
Oct 30 05:51:07.247963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:07.247933    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:08.081489 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:08.081453    5250 generic.go:356] "Generic (PLEG): container finished" podID="19e9dae2-fad2-4c83-a068-567aeabac096" containerID="357505cfcf2bdb8a8ab9bffece83c2c4a47229e0037689f1c1d9fe80287f1285" exitCode=0
Oct 30 05:51:08.081901 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:08.081540    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j" event={"ID":"19e9dae2-fad2-4c83-a068-567aeabac096","Type":"ContainerDied","Data":"357505cfcf2bdb8a8ab9bffece83c2c4a47229e0037689f1c1d9fe80287f1285"}
Oct 30 05:51:08.859427 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:08.859391    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:09.240085 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.240062    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:09.365547 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.365462    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-xks-ca\") pod \"19e9dae2-fad2-4c83-a068-567aeabac096\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") "
Oct 30 05:51:09.365705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.365677    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-openshift-ca\") pod \"19e9dae2-fad2-4c83-a068-567aeabac096\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") "
Oct 30 05:51:09.365757 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.365722    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e9dae2-fad2-4c83-a068-567aeabac096-kube-api-access-wjtgd\" (UniqueName: \"kubernetes.io/projected/19e9dae2-fad2-4c83-a068-567aeabac096-kube-api-access-wjtgd\") pod \"19e9dae2-fad2-4c83-a068-567aeabac096\" (UID: \"19e9dae2-fad2-4c83-a068-567aeabac096\") "
Oct 30 05:51:09.366072 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.366046    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-openshift-ca" pod "19e9dae2-fad2-4c83-a068-567aeabac096" (UID: "19e9dae2-fad2-4c83-a068-567aeabac096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:51:09.366406 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.366351    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:51:09.367765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.367739    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-xks-ca" pod "19e9dae2-fad2-4c83-a068-567aeabac096" (UID: "19e9dae2-fad2-4c83-a068-567aeabac096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:51:09.367884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.367868    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e9dae2-fad2-4c83-a068-567aeabac096-kube-api-access-wjtgd" pod "19e9dae2-fad2-4c83-a068-567aeabac096" (UID: "19e9dae2-fad2-4c83-a068-567aeabac096"). InnerVolumeSpecName "kube-api-access-wjtgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:51:09.467296 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.467267    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjtgd\" (UniqueName: \"kubernetes.io/projected/19e9dae2-fad2-4c83-a068-567aeabac096-kube-api-access-wjtgd\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:51:09.467296 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:09.467292    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e9dae2-fad2-4c83-a068-567aeabac096-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:51:10.096382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:10.096355    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"
Oct 30 05:51:10.096382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:10.096371    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j" event={"ID":"19e9dae2-fad2-4c83-a068-567aeabac096","Type":"ContainerDied","Data":"89437e0ea72c4608aeb76dc940bfa63ea4d339c9e402c5795ea55d435e27fdbe"}
Oct 30 05:51:10.096596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:10.096416    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89437e0ea72c4608aeb76dc940bfa63ea4d339c9e402c5795ea55d435e27fdbe"
Oct 30 05:51:10.551696 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:10.551662    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:12.314397 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:12.314366    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:14.615880 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:14.615831    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:51:14.617719 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:14.617687    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:51:14.619378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:14.619350    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:51:14.762214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:14.738639    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:51:14.809204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:14.797289    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:51:14.818638 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:14.818300    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:14.970551 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:14.970510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:16.026915 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:16.026879    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:16.150919 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:16.150886    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"]
Oct 30 05:51:16.185684 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:16.185649    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"]
Oct 30 05:51:16.332845 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:16.332764    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:16.475059 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:16.475031    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:18.024254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.024169    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:18.276478 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.276393    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"]
Oct 30 05:51:18.277526 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.277495    5250 cpu_manager.go:414] "containerMap: removing container" podUID="19e9dae2-fad2-4c83-a068-567aeabac096" containerName="revoke-keys"
Oct 30 05:51:18.277526 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.277523    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e9dae2-fad2-4c83-a068-567aeabac096" containerName="revoke-keys"
Oct 30 05:51:18.277709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.277695    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e9dae2-fad2-4c83-a068-567aeabac096" containerName="revoke-keys"
Oct 30 05:51:18.282514 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.282495    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.297229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.297196    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"]
Oct 30 05:51:18.308499 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.308453    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.319541 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.319503    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"]
Oct 30 05:51:18.335727 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.335684    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16313d3-95c4-4f28-afb9-d577320359e7-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.338200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.338155    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"]
Oct 30 05:51:18.338406 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.338165    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.338527 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.338506    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.338601 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.338542    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8gq\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-kube-api-access-mb8gq\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.338662 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.338638    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.338748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.338730    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.338798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.338773    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.339135 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339113    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.339271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339151    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.339271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339235    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.339382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339334    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.339382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339374    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.339485 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339451    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.339539 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339529    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16313d3-95c4-4f28-afb9-d577320359e7-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.339603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339586    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.339660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339619    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.339660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339644    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zzc7\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-kube-api-access-9zzc7\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.339760 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.339679    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.441503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441465    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.441697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441541    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.441697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441609    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16313d3-95c4-4f28-afb9-d577320359e7-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.441697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441657    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.441697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441682    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.441962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441709    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zzc7\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-kube-api-access-9zzc7\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.441962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441743    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.441962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441850    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16313d3-95c4-4f28-afb9-d577320359e7-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.441962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441884    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.441962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441909    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.441962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441936    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8gq\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-kube-api-access-mb8gq\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441971    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.441983    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442026    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442065    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442103    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442134    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442144    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442200    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.442279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442242    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.442718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442343    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.442718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442347    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.442718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442420    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.443734 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.442899    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16313d3-95c4-4f28-afb9-d577320359e7-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.443734 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.443152    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.444262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.444219    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.444386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.444350    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.444913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.444887    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.446451 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.446368    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.448351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.447242    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16313d3-95c4-4f28-afb9-d577320359e7-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.448541 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.448500    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.448805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.448780    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.469255 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.469213    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.469643 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.469620    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8gq\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-kube-api-access-mb8gq\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.469817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.469795    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.469861 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.469823    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zzc7\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-kube-api-access-9zzc7\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.603912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.603830    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:18.645517 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.645479    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:18.892209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.888410    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:18.972935 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.972894    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"]
Oct 30 05:51:18.986761 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:51:18.986729    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16313d3_95c4_4f28_afb9_d577320359e7.slice/crio-88cab3a152218cba2903eec96aa48c7c09cdba50affa1852bcdabc0bf780719f WatchSource:0}: Error finding container 88cab3a152218cba2903eec96aa48c7c09cdba50affa1852bcdabc0bf780719f: Status 404 returned error can't find the container with id 88cab3a152218cba2903eec96aa48c7c09cdba50affa1852bcdabc0bf780719f
Oct 30 05:51:18.991400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.991345    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:18.991504 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.991458    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:18.991543 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:18.991500    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:19.014605 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.014576    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"]
Oct 30 05:51:19.017643 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.017615    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:19.017738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.017678    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:19.017738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.017705    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:19.276964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.276864    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:19.322610 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.322579    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" event={"ID":"de63ae29-2a4a-44b4-b0fc-63201d94bd87","Type":"ContainerStarted","Data":"bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4"}
Oct 30 05:51:19.322875 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.322852    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" event={"ID":"de63ae29-2a4a-44b4-b0fc-63201d94bd87","Type":"ContainerStarted","Data":"b0c2acefbf6cd8b453e4d40abd2927fdf48350555bc0b47dab2d0ccd167b9761"}
Oct 30 05:51:19.325428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.324994    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" event={"ID":"e16313d3-95c4-4f28-afb9-d577320359e7","Type":"ContainerStarted","Data":"f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21"}
Oct 30 05:51:19.325428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.325048    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" event={"ID":"e16313d3-95c4-4f28-afb9-d577320359e7","Type":"ContainerStarted","Data":"88cab3a152218cba2903eec96aa48c7c09cdba50affa1852bcdabc0bf780719f"}
Oct 30 05:51:19.371410 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.371348    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" podStartSLOduration=1.371328701 podStartE2EDuration="1.371328701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:19.370835677 +0000 UTC m=+1194.201872898" watchObservedRunningTime="2026-10-30 05:51:19.371328701 +0000 UTC m=+1194.202365919"
Oct 30 05:51:19.480447 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.480385    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podStartSLOduration=1.480365968 podStartE2EDuration="1.480365968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:19.478431876 +0000 UTC m=+1194.309469094" watchObservedRunningTime="2026-10-30 05:51:19.480365968 +0000 UTC m=+1194.311403186"
Oct 30 05:51:19.606061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.605976    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:19.647603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.647566    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:19.740751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:19.740717    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:20.038130 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:20.038097    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:20.390776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:20.390731    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:20.621780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:20.621730    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 05:51:20.630202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:20.627253    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 05:51:20.649417 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:20.649315    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 05:51:20.649417 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:20.649376    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 05:51:20.835890 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:20.835855    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:21.188307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:21.188249    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp" podUID="42b0bac3-2617-4505-b1ab-6f49cbd83244" containerName="payload-pre-processing" containerID="cri-o://437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8" gracePeriod=25
Oct 30 05:51:21.242306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:21.240273    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp" podUID="504727f8-a865-4093-88db-840e38f8fad7" containerName="payload-processing" containerID="cri-o://d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1" gracePeriod=25
Oct 30 05:51:21.492031 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:21.491954    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:21.616896 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:21.616863    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 05:51:21.617630 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:21.617596    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 05:51:21.650118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:21.650082    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 05:51:21.650415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:21.650387    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 05:51:21.931221 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:21.924820    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:22.150001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.149970    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:51:22.277207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.270029    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:22.343544 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.339480    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504727f8-a865-4093-88db-840e38f8fad7-kube-api-access-rzbrv\" (UniqueName: \"kubernetes.io/projected/504727f8-a865-4093-88db-840e38f8fad7-kube-api-access-rzbrv\") pod \"504727f8-a865-4093-88db-840e38f8fad7\" (UID: \"504727f8-a865-4093-88db-840e38f8fad7\") "
Oct 30 05:51:22.343544 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.339772    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504727f8-a865-4093-88db-840e38f8fad7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504727f8-a865-4093-88db-840e38f8fad7-plugins-config-volume\") pod \"504727f8-a865-4093-88db-840e38f8fad7\" (UID: \"504727f8-a865-4093-88db-840e38f8fad7\") "
Oct 30 05:51:22.392223 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.387113    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504727f8-a865-4093-88db-840e38f8fad7-plugins-config-volume" pod "504727f8-a865-4093-88db-840e38f8fad7" (UID: "504727f8-a865-4093-88db-840e38f8fad7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:51:22.434311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.433345    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504727f8-a865-4093-88db-840e38f8fad7-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:51:22.496914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.496876    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504727f8-a865-4093-88db-840e38f8fad7-kube-api-access-rzbrv" pod "504727f8-a865-4093-88db-840e38f8fad7" (UID: "504727f8-a865-4093-88db-840e38f8fad7"). InnerVolumeSpecName "kube-api-access-rzbrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:51:22.560202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.552029    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzbrv\" (UniqueName: \"kubernetes.io/projected/504727f8-a865-4093-88db-840e38f8fad7-kube-api-access-rzbrv\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:51:22.560202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.555869    5250 generic.go:356] "Generic (PLEG): container finished" podID="504727f8-a865-4093-88db-840e38f8fad7" containerID="d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1" exitCode=0
Oct 30 05:51:22.560202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.556062    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp" event={"ID":"504727f8-a865-4093-88db-840e38f8fad7","Type":"ContainerDied","Data":"d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1"}
Oct 30 05:51:22.560202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.556121    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp" event={"ID":"504727f8-a865-4093-88db-840e38f8fad7","Type":"ContainerDied","Data":"4afc94b721e5a102bda20251c0ad62dab0d688b8b682db3d6c8526ab41fa5c80"}
Oct 30 05:51:22.560202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.556152    5250 scope.go:122] "RemoveContainer" containerID="d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1"
Oct 30 05:51:22.560202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.556508    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"
Oct 30 05:51:22.560202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.557681    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:51:22.563683 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.563644    5250 generic.go:356] "Generic (PLEG): container finished" podID="42b0bac3-2617-4505-b1ab-6f49cbd83244" containerID="437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8" exitCode=0
Oct 30 05:51:22.564059 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.564024    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp" event={"ID":"42b0bac3-2617-4505-b1ab-6f49cbd83244","Type":"ContainerDied","Data":"437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8"}
Oct 30 05:51:22.564263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.564231    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp" event={"ID":"42b0bac3-2617-4505-b1ab-6f49cbd83244","Type":"ContainerDied","Data":"689ce4a431619ee3ff43f77c23c312c01bbaf0f9577c8a6b28909739dce63cbf"}
Oct 30 05:51:22.617929 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.604731    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 05:51:22.617929 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.604784    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 05:51:22.687271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.665016    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b0bac3-2617-4505-b1ab-6f49cbd83244-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b0bac3-2617-4505-b1ab-6f49cbd83244-plugins-config-volume\") pod \"42b0bac3-2617-4505-b1ab-6f49cbd83244\" (UID: \"42b0bac3-2617-4505-b1ab-6f49cbd83244\") "
Oct 30 05:51:22.717227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.713350    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b0bac3-2617-4505-b1ab-6f49cbd83244-kube-api-access-dnkz6\" (UniqueName: \"kubernetes.io/projected/42b0bac3-2617-4505-b1ab-6f49cbd83244-kube-api-access-dnkz6\") pod \"42b0bac3-2617-4505-b1ab-6f49cbd83244\" (UID: \"42b0bac3-2617-4505-b1ab-6f49cbd83244\") "
Oct 30 05:51:22.744203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.667703    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b0bac3-2617-4505-b1ab-6f49cbd83244-plugins-config-volume" pod "42b0bac3-2617-4505-b1ab-6f49cbd83244" (UID: "42b0bac3-2617-4505-b1ab-6f49cbd83244"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:51:22.744203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.675854    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 05:51:22.744203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.722135    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 05:51:22.746813 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.746776    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"]
Oct 30 05:51:22.800407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.799488    5250 scope.go:122] "RemoveContainer" containerID="d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1"
Oct 30 05:51:22.804369 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:51:22.804336    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1\": container with ID starting with d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1 not found: ID does not exist" containerID="d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1"
Oct 30 05:51:22.804642 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.804616    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1"} err="failed to get container status \"d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1\": rpc error: code = NotFound desc = could not find container \"d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1\": container with ID starting with d1c9c72cc5d0ff7369a1ac4be6e74076e2c5312059436933d49bbdf7ebc2a1a1 not found: ID does not exist"
Oct 30 05:51:22.804763 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.804752    5250 scope.go:122] "RemoveContainer" containerID="437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8"
Oct 30 05:51:22.826212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.819314    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b0bac3-2617-4505-b1ab-6f49cbd83244-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:51:22.826474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.826407    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b0bac3-2617-4505-b1ab-6f49cbd83244-kube-api-access-dnkz6" pod "42b0bac3-2617-4505-b1ab-6f49cbd83244" (UID: "42b0bac3-2617-4505-b1ab-6f49cbd83244"). InnerVolumeSpecName "kube-api-access-dnkz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:51:22.846306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.844009    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3cc280cf-75c9c49fd8-xlssp"]
Oct 30 05:51:22.894506 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.894374    5250 scope.go:122] "RemoveContainer" containerID="437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8"
Oct 30 05:51:22.895215 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:51:22.894774    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8\": container with ID starting with 437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8 not found: ID does not exist" containerID="437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8"
Oct 30 05:51:22.895215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.894809    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8"} err="failed to get container status \"437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8\": rpc error: code = NotFound desc = could not find container \"437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8\": container with ID starting with 437772eade8660e58548d11dd3a333f194aac49513c0f50eb06de41180b351d8 not found: ID does not exist"
Oct 30 05:51:22.911201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.904506    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:22.932195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:22.928063    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnkz6\" (UniqueName: \"kubernetes.io/projected/42b0bac3-2617-4505-b1ab-6f49cbd83244-kube-api-access-dnkz6\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:51:23.143493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.143449    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:23.208203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.203454    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"]
Oct 30 05:51:23.250043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.249948    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3cc280cf-bwl6j"]
Oct 30 05:51:23.327203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.326416    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:23.341145 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.341107    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e9dae2-fad2-4c83-a068-567aeabac096" path="/var/lib/kubelet/pods/19e9dae2-fad2-4c83-a068-567aeabac096/volumes"
Oct 30 05:51:23.342921 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.342899    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504727f8-a865-4093-88db-840e38f8fad7" path="/var/lib/kubelet/pods/504727f8-a865-4093-88db-840e38f8fad7/volumes"
Oct 30 05:51:23.495260 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.490872    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:23.607203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.606387    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 05:51:23.607203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.606447    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 05:51:23.654205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.653719    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 05:51:23.654205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.653789    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 05:51:23.686283 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.682055    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"
Oct 30 05:51:23.791451 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.788226    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"]
Oct 30 05:51:23.831204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:23.827526    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3cc280cf-6f55f7d96c-qslgp"]
Oct 30 05:51:24.416999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:24.416967    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:24.578344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:24.578310    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:24.609811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:24.609778    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 05:51:24.610423 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:24.610390    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 05:51:24.647412 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:24.647385    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 05:51:24.647594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:24.647576    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 05:51:25.318764 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:25.318399    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b0bac3-2617-4505-b1ab-6f49cbd83244" path="/var/lib/kubelet/pods/42b0bac3-2617-4505-b1ab-6f49cbd83244/volumes"
Oct 30 05:51:25.577263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:25.554736    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:51:25.593221 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:25.587957    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:51:25.604949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:25.604913    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 05:51:25.605134 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:25.604961    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 05:51:25.647236 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:25.647167    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 05:51:25.647236 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:25.647237    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 05:51:25.921535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:25.921495    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:26.604924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:26.604887    5250 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 05:51:26.605207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:26.604967    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 05:51:27.091986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.091952    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:27.092595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.092534    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:27.093397 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.093373    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:51:27.384913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.384873    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:27.533981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.533902    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:27.608595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.608568    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:27.608997 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.608980    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:27.609661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.609642    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:51:27.891155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:27.891122    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:28.036783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:28.036751    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:29.188884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.188854    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:29.323637 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.323607    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"]
Oct 30 05:51:29.324206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.324172    5250 cpu_manager.go:414] "containerMap: removing container" podUID="504727f8-a865-4093-88db-840e38f8fad7" containerName="payload-processing"
Oct 30 05:51:29.324206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.324199    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504727f8-a865-4093-88db-840e38f8fad7" containerName="payload-processing"
Oct 30 05:51:29.324286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.324234    5250 cpu_manager.go:414] "containerMap: removing container" podUID="42b0bac3-2617-4505-b1ab-6f49cbd83244" containerName="payload-pre-processing"
Oct 30 05:51:29.324286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.324239    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b0bac3-2617-4505-b1ab-6f49cbd83244" containerName="payload-pre-processing"
Oct 30 05:51:29.324361 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.324340    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b0bac3-2617-4505-b1ab-6f49cbd83244" containerName="payload-pre-processing"
Oct 30 05:51:29.324361 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.324352    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="504727f8-a865-4093-88db-840e38f8fad7" containerName="payload-processing"
Oct 30 05:51:29.329715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.329695    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:29.339889 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.339862    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:29.348158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.348136    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"]
Oct 30 05:51:29.356992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.356968    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3224b039\""
Oct 30 05:51:29.356992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.356978    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3224b039-dockercfg-spvqj\""
Oct 30 05:51:29.389515 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.389479    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmdl\" (UniqueName: \"kubernetes.io/projected/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-kube-api-access-rkmdl\") pod \"payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr\" (UID: \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:29.389973 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.389954    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr\" (UID: \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:29.470511 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.470405    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"]
Oct 30 05:51:29.479508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.479476    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"]
Oct 30 05:51:29.479684 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.479606    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:29.491539 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.491508    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr\" (UID: \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:29.491767 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.491745    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmdl\" (UniqueName: \"kubernetes.io/projected/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-kube-api-access-rkmdl\") pod \"payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr\" (UID: \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:29.492233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.492214    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr\" (UID: \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:29.516671 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.516646    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmdl\" (UniqueName: \"kubernetes.io/projected/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-kube-api-access-rkmdl\") pod \"payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr\" (UID: \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:29.604263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.604225    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8854e3da-6a3e-498d-b447-83ac72033214-plugins-config-volume\") pod \"payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85\" (UID: \"8854e3da-6a3e-498d-b447-83ac72033214\") " pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:29.604723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.604691    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9lj\" (UniqueName: \"kubernetes.io/projected/8854e3da-6a3e-498d-b447-83ac72033214-kube-api-access-xq9lj\") pod \"payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85\" (UID: \"8854e3da-6a3e-498d-b447-83ac72033214\") " pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:29.644326 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.644283    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:29.708166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.706493    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9lj\" (UniqueName: \"kubernetes.io/projected/8854e3da-6a3e-498d-b447-83ac72033214-kube-api-access-xq9lj\") pod \"payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85\" (UID: \"8854e3da-6a3e-498d-b447-83ac72033214\") " pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:29.708166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.706799    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8854e3da-6a3e-498d-b447-83ac72033214-plugins-config-volume\") pod \"payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85\" (UID: \"8854e3da-6a3e-498d-b447-83ac72033214\") " pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:29.719768 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.719736    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8854e3da-6a3e-498d-b447-83ac72033214-plugins-config-volume\") pod \"payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85\" (UID: \"8854e3da-6a3e-498d-b447-83ac72033214\") " pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:29.749157 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.749094    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9lj\" (UniqueName: \"kubernetes.io/projected/8854e3da-6a3e-498d-b447-83ac72033214-kube-api-access-xq9lj\") pod \"payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85\" (UID: \"8854e3da-6a3e-498d-b447-83ac72033214\") " pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:29.797336 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.797300    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:29.969013 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:29.968963    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"]
Oct 30 05:51:30.116264 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:51:30.116231    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8854e3da_6a3e_498d_b447_83ac72033214.slice/crio-5b953611526aaca2a70391d44d461f6439ec6bc2d36b0987ac318161350e63bd WatchSource:0}: Error finding container 5b953611526aaca2a70391d44d461f6439ec6bc2d36b0987ac318161350e63bd: Status 404 returned error can't find the container with id 5b953611526aaca2a70391d44d461f6439ec6bc2d36b0987ac318161350e63bd
Oct 30 05:51:30.118261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.118233    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"]
Oct 30 05:51:30.779005 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.778961    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr" event={"ID":"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6","Type":"ContainerStarted","Data":"c59ac8813c5a46a3f9b3fd9be50e3ca99cd974432d03076c73419fd464407cee"}
Oct 30 05:51:30.779457 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.779160    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:30.779457 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.779229    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr" event={"ID":"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6","Type":"ContainerStarted","Data":"b9962912a41624e2a3748fb2510a193bcc890b4a3b24b6ce095120fec4b90958"}
Oct 30 05:51:30.780793 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.780771    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85" event={"ID":"8854e3da-6a3e-498d-b447-83ac72033214","Type":"ContainerStarted","Data":"1dc007f45c578c2560950e41fa804b5474e8340c2ec7a9fe57d66507d20b1124"}
Oct 30 05:51:30.780918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.780814    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85" event={"ID":"8854e3da-6a3e-498d-b447-83ac72033214","Type":"ContainerStarted","Data":"5b953611526aaca2a70391d44d461f6439ec6bc2d36b0987ac318161350e63bd"}
Oct 30 05:51:30.780918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.780858    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:30.866294 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.866246    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr" podStartSLOduration=1.8662338520000001 podStartE2EDuration="1.866233852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:30.823265479 +0000 UTC m=+1205.654302696" watchObservedRunningTime="2026-10-30 05:51:30.866233852 +0000 UTC m=+1205.697271074"
Oct 30 05:51:30.868574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.868542    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85" podStartSLOduration=1.868533125 podStartE2EDuration="1.868533125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:30.865583923 +0000 UTC m=+1205.696621140" watchObservedRunningTime="2026-10-30 05:51:30.868533125 +0000 UTC m=+1205.699570341"
Oct 30 05:51:30.885639 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:30.885618    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:31.048452 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:31.048367    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:32.205652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:32.205617    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:32.374286 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:51:32.374245    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:51:32.645390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:32.645353    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:34.569075 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:34.569037    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:36.054765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:36.054726    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:36.719309 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:36.719258    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:37.613061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:37.613023    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:39.383728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:39.383637    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:40.948579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:40.948530    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:41.791473 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:41.791444    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:51:41.791683 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:41.791530    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:51:42.503368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:42.503333    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:43.379204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:43.368630    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:44.030497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:44.030468    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:44.184227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:44.174267    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:44.326156 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:44.326060    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:44.893012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:44.892965    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:45.033448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.033418    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:45.077173 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.077140    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"]
Oct 30 05:51:45.081962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.081929    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.109445 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.109416    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 05:51:45.124073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.122795    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 05:51:45.124073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.122866    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h6vpc\""
Oct 30 05:51:45.124073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.122911    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 05:51:45.127749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.127724    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"]
Oct 30 05:51:45.177855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.177303    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.177855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.177397    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b151f1-9236-4fdf-bc13-2d670c450e5a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.177855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.177438    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5cw\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-kube-api-access-vp5cw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.177855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.177515    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.177855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.177537    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.177855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.177556    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.177855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.177652    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.278663 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.278628    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.278827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.278709    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.278827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.278778    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b151f1-9236-4fdf-bc13-2d670c450e5a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.278974 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.278831    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5cw\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-kube-api-access-vp5cw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.278974 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.278938    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.279071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.278981    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.279071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.279012    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.280260 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.280235    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.282021 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.281978    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b151f1-9236-4fdf-bc13-2d670c450e5a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.282263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.282239    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.283078 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.283055    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.283412 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.283382    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.306472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.306437    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5cw\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-kube-api-access-vp5cw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.323418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.323385    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lp26r\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.417385 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.417354    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:45.607085 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.607050    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:45.835913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.835876    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"]
Oct 30 05:51:45.836920 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:51:45.836868    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b151f1_9236_4fdf_bc13_2d670c450e5a.slice/crio-fdc0ac41eeb690b17b1dd7627d64ec0068e208c617cdfddc92396b002541f2ef WatchSource:0}: Error finding container fdc0ac41eeb690b17b1dd7627d64ec0068e208c617cdfddc92396b002541f2ef: Status 404 returned error can't find the container with id fdc0ac41eeb690b17b1dd7627d64ec0068e208c617cdfddc92396b002541f2ef
Oct 30 05:51:45.930027 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:45.929983    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r" event={"ID":"e9b151f1-9236-4fdf-bc13-2d670c450e5a","Type":"ContainerStarted","Data":"fdc0ac41eeb690b17b1dd7627d64ec0068e208c617cdfddc92396b002541f2ef"}
Oct 30 05:51:46.931386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:46.931345    5250 patch_prober.go:29] interesting pod/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:51:46.931809 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:46.931394    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:51:47.090348 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:47.090284    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:48.575428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.575399    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:48.575723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.575458    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:48.628669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.628637    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:48.958683 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.958650    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r" event={"ID":"e9b151f1-9236-4fdf-bc13-2d670c450e5a","Type":"ContainerStarted","Data":"bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c"}
Oct 30 05:51:48.958894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.958734    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:48.960374 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.960347    5250 generic.go:356] "Generic (PLEG): container finished" podID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerID="5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a" exitCode=0
Oct 30 05:51:48.960491 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.960413    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" event={"ID":"6ab10583-5b3b-4092-83e0-beee39ba10a7","Type":"ContainerDied","Data":"5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a"}
Oct 30 05:51:48.960828 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.960807    5250 scope.go:122] "RemoveContainer" containerID="5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a"
Oct 30 05:51:48.962381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.962358    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:48.962452 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.962425    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:48.962491 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:48.962456    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:51:49.036708 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:49.036652    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r" podStartSLOduration=1.301778919 podStartE2EDuration="4.036631971s" totalImagesPullingTime="2.734853052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:49.019851636 +0000 UTC m=+1223.850888853" watchObservedRunningTime="2026-10-30 05:51:49.036631971 +0000 UTC m=+1223.867669189"
Oct 30 05:51:49.994081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:49.994044    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" event={"ID":"6ab10583-5b3b-4092-83e0-beee39ba10a7","Type":"ContainerStarted","Data":"772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e"}
Oct 30 05:51:49.996024 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:49.996001    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 05:51:50.053234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:50.053206    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:50.420767 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:50.420739    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:51:50.420981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:50.420959    5250 patch_prober.go:29] interesting pod/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 05:51:50.421030 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:50.421005    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 05:51:51.421157 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:51.421124    5250 patch_prober.go:29] interesting pod/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 05:51:51.421578 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:51.421197    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 05:51:51.684259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:51.683653    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:52.198453 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:52.198418    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:52.421652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:52.421606    5250 patch_prober.go:29] interesting pod/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 05:51:52.422330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:52.422283    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 05:51:53.308311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:53.308254    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:53.421405 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:53.421366    5250 patch_prober.go:29] interesting pod/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 05:51:53.421628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:53.421432    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 05:51:54.420447 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:54.420406    5250 patch_prober.go:29] interesting pod/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 05:51:54.420865 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:54.420471    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 05:51:54.978206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:54.974675    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:55.424621 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:55.424589    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:51:55.425049 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:55.425026    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:51:55.425798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:55.425781    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:51:56.612735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:56.612697    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:58.174086 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:58.174048    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:59.781321 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:51:59.781278    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:01.588984 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:01.588946    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:01.739393 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:01.739348    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:02.374351 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:52:02.374310    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:52:03.369789 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:03.369740    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"]
Oct 30 05:52:03.370657 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:03.370601    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy" containerID="cri-o://772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e" gracePeriod=30
Oct 30 05:52:03.427452 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:03.427424    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:03.588004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:03.587963    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:04.379957 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:04.379925    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:04.652401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:04.652364    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:04.846154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:04.846113    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:05.541435 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:05.541399    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:05.711205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:05.710374    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:05.882117 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:05.881990    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:07.954897 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:07.954798    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:08.652503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:08.651290    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:52:08.977470 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:08.977439    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:52:09.153608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.153574    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-data\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.153807 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.153695    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-podinfo\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.153807 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.153756    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-envoy\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.153951 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.153838    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-data" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:09.153951 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.153870    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab10583-5b3b-4092-83e0-beee39ba10a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab10583-5b3b-4092-83e0-beee39ba10a7-istiod-ca-cert\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.153951 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.153903    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-kube-api-access-xgbjs\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-kube-api-access-xgbjs\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.153951 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.153940    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-token\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-token\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.154165 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.153996    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-credential-socket\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.154292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154166    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-certs\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.154292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154228    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab10583-5b3b-4092-83e0-beee39ba10a7-istiod-ca-cert" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:52:09.154292 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154283    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-socket\") pod \"6ab10583-5b3b-4092-83e0-beee39ba10a7\" (UID: \"6ab10583-5b3b-4092-83e0-beee39ba10a7\") "
Oct 30 05:52:09.154524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154471    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-credential-socket" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:09.154524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154508    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-socket" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:09.154884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154862    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.154884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154885    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.155075 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154898    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab10583-5b3b-4092-83e0-beee39ba10a7-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.155075 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.154909    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.155224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.155152    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-certs" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:09.158323 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.158297    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-podinfo" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:52:09.158479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.158303    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-envoy" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:09.158479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.158361    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-kube-api-access-xgbjs" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "kube-api-access-xgbjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:09.158626 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.158374    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-token" pod "6ab10583-5b3b-4092-83e0-beee39ba10a7" (UID: "6ab10583-5b3b-4092-83e0-beee39ba10a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:09.238357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.238265    5250 generic.go:356] "Generic (PLEG): container finished" podID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerID="772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e" exitCode=0
Oct 30 05:52:09.238357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.238323    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"
Oct 30 05:52:09.238357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.238323    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" event={"ID":"6ab10583-5b3b-4092-83e0-beee39ba10a7","Type":"ContainerDied","Data":"772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e"}
Oct 30 05:52:09.238585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.238413    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65" event={"ID":"6ab10583-5b3b-4092-83e0-beee39ba10a7","Type":"ContainerDied","Data":"b648d33edf24467de3c621f29d2aa2143fc9f603e2c9da0b83af8d3cd39aaa5d"}
Oct 30 05:52:09.238585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.238436    5250 scope.go:122] "RemoveContainer" containerID="772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e"
Oct 30 05:52:09.249891 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.249873    5250 scope.go:122] "RemoveContainer" containerID="5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a"
Oct 30 05:52:09.256349 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.256323    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.256446 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.256352    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.256446 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.256369    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgbjs\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-kube-api-access-xgbjs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.256446 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.256383    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab10583-5b3b-4092-83e0-beee39ba10a7-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.256446 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.256396    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab10583-5b3b-4092-83e0-beee39ba10a7-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:09.260520 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.260504    5250 scope.go:122] "RemoveContainer" containerID="772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e"
Oct 30 05:52:09.260789 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:52:09.260768    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e\": container with ID starting with 772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e not found: ID does not exist" containerID="772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e"
Oct 30 05:52:09.260900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.260798    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e"} err="failed to get container status \"772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e\": rpc error: code = NotFound desc = could not find container \"772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e\": container with ID starting with 772ddc3de818739fabe5fa008a0a037c023ff513ed523ed534d7c02159cb718e not found: ID does not exist"
Oct 30 05:52:09.260900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.260815    5250 scope.go:122] "RemoveContainer" containerID="5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a"
Oct 30 05:52:09.261072 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:52:09.261056    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a\": container with ID starting with 5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a not found: ID does not exist" containerID="5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a"
Oct 30 05:52:09.261112 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.261079    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a"} err="failed to get container status \"5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a\": rpc error: code = NotFound desc = could not find container \"5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a\": container with ID starting with 5aba31b547017818960622da2b5ee01b6e97da00e383ca68daad75130859878a not found: ID does not exist"
Oct 30 05:52:09.297680 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.297652    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"]
Oct 30 05:52:09.301237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:09.301218    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-19e33488-openshift-default-6d958965c5-r5t65"]
Oct 30 05:52:10.003534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:10.003493    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:11.281481 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:11.281452    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" path="/var/lib/kubelet/pods/6ab10583-5b3b-4092-83e0-beee39ba10a7/volumes"
Oct 30 05:52:11.608314 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:11.608213    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:12.234975 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:12.234935    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:12.380615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:12.380584    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:12.565373 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:12.565150    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:12.966780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:12.966745    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"]
Oct 30 05:52:13.001034 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:13.001002    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"]
Oct 30 05:52:13.155945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:13.155912    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:13.297079 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:13.296998    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:14.971068 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:14.971024    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:15.110544 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:15.110494    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:15.576381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:15.576350    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:16.216935 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:16.216898    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:16.629687 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:16.629595    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:17.393202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:17.389811    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:17.635246 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:17.635154    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:17.899442 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:17.899401    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:17.967279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:17.967229    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85" podUID="8854e3da-6a3e-498d-b447-83ac72033214" containerName="payload-processing" containerID="cri-o://1dc007f45c578c2560950e41fa804b5474e8340c2ec7a9fe57d66507d20b1124" gracePeriod=25
Oct 30 05:52:18.002214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.002169    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr" podUID="1beb1bd2-dcc6-4323-8c68-87c1960cb0b6" containerName="payload-pre-processing" containerID="cri-o://c59ac8813c5a46a3f9b3fd9be50e3ca99cd974432d03076c73419fd464407cee" gracePeriod=25
Oct 30 05:52:18.087498 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.087468    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:18.327242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.324540    5250 generic.go:356] "Generic (PLEG): container finished" podID="8854e3da-6a3e-498d-b447-83ac72033214" containerID="1dc007f45c578c2560950e41fa804b5474e8340c2ec7a9fe57d66507d20b1124" exitCode=0
Oct 30 05:52:18.327242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.324747    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85" event={"ID":"8854e3da-6a3e-498d-b447-83ac72033214","Type":"ContainerDied","Data":"1dc007f45c578c2560950e41fa804b5474e8340c2ec7a9fe57d66507d20b1124"}
Oct 30 05:52:18.327242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.326792    5250 generic.go:356] "Generic (PLEG): container finished" podID="1beb1bd2-dcc6-4323-8c68-87c1960cb0b6" containerID="c59ac8813c5a46a3f9b3fd9be50e3ca99cd974432d03076c73419fd464407cee" exitCode=0
Oct 30 05:52:18.327242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.326842    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr" event={"ID":"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6","Type":"ContainerDied","Data":"c59ac8813c5a46a3f9b3fd9be50e3ca99cd974432d03076c73419fd464407cee"}
Oct 30 05:52:18.384695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.384602    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:18.424432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.424408    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:52:18.459566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.459544    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:52:18.512883 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.512782    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8854e3da-6a3e-498d-b447-83ac72033214-kube-api-access-xq9lj\" (UniqueName: \"kubernetes.io/projected/8854e3da-6a3e-498d-b447-83ac72033214-kube-api-access-xq9lj\") pod \"8854e3da-6a3e-498d-b447-83ac72033214\" (UID: \"8854e3da-6a3e-498d-b447-83ac72033214\") "
Oct 30 05:52:18.512883 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.512846    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-plugins-config-volume\") pod \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\" (UID: \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\") "
Oct 30 05:52:18.513124 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.513018    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8854e3da-6a3e-498d-b447-83ac72033214-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8854e3da-6a3e-498d-b447-83ac72033214-plugins-config-volume\") pod \"8854e3da-6a3e-498d-b447-83ac72033214\" (UID: \"8854e3da-6a3e-498d-b447-83ac72033214\") "
Oct 30 05:52:18.513124 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.513046    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-kube-api-access-rkmdl\" (UniqueName: \"kubernetes.io/projected/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-kube-api-access-rkmdl\") pod \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\" (UID: \"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6\") "
Oct 30 05:52:18.513513 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.513394    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-plugins-config-volume" pod "1beb1bd2-dcc6-4323-8c68-87c1960cb0b6" (UID: "1beb1bd2-dcc6-4323-8c68-87c1960cb0b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:52:18.513513 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.513450    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8854e3da-6a3e-498d-b447-83ac72033214-plugins-config-volume" pod "8854e3da-6a3e-498d-b447-83ac72033214" (UID: "8854e3da-6a3e-498d-b447-83ac72033214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:52:18.514019 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.513992    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:18.514019 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.514019    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8854e3da-6a3e-498d-b447-83ac72033214-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:18.516064 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.516038    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8854e3da-6a3e-498d-b447-83ac72033214-kube-api-access-xq9lj" pod "8854e3da-6a3e-498d-b447-83ac72033214" (UID: "8854e3da-6a3e-498d-b447-83ac72033214"). InnerVolumeSpecName "kube-api-access-xq9lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:18.516227 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.516077    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-kube-api-access-rkmdl" pod "1beb1bd2-dcc6-4323-8c68-87c1960cb0b6" (UID: "1beb1bd2-dcc6-4323-8c68-87c1960cb0b6"). InnerVolumeSpecName "kube-api-access-rkmdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:18.615346 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.615315    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9lj\" (UniqueName: \"kubernetes.io/projected/8854e3da-6a3e-498d-b447-83ac72033214-kube-api-access-xq9lj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:18.615346 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:18.615343    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmdl\" (UniqueName: \"kubernetes.io/projected/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6-kube-api-access-rkmdl\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:19.033155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.033122    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:19.200965 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.200926    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:19.335882 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.335851    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"
Oct 30 05:52:19.336055 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.335877    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr" event={"ID":"1beb1bd2-dcc6-4323-8c68-87c1960cb0b6","Type":"ContainerDied","Data":"b9962912a41624e2a3748fb2510a193bcc890b4a3b24b6ce095120fec4b90958"}
Oct 30 05:52:19.336055 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.335950    5250 scope.go:122] "RemoveContainer" containerID="c59ac8813c5a46a3f9b3fd9be50e3ca99cd974432d03076c73419fd464407cee"
Oct 30 05:52:19.338897 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.338863    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85" event={"ID":"8854e3da-6a3e-498d-b447-83ac72033214","Type":"ContainerDied","Data":"5b953611526aaca2a70391d44d461f6439ec6bc2d36b0987ac318161350e63bd"}
Oct 30 05:52:19.339027 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.338974    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"
Oct 30 05:52:19.351490 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.351465    5250 scope.go:122] "RemoveContainer" containerID="1dc007f45c578c2560950e41fa804b5474e8340c2ec7a9fe57d66507d20b1124"
Oct 30 05:52:19.398836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.398806    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"]
Oct 30 05:52:19.406076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.403978    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3224b039-d9f4c7d98-2hl85"]
Oct 30 05:52:19.461803 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.461772    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"]
Oct 30 05:52:19.466199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.466151    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3224b039-794656d95f-td9cr"]
Oct 30 05:52:19.554898 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.554817    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:19.715432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:19.715389    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:20.062456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:20.061835    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:20.219050 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:20.219015    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:21.256449 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:21.256390    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:21.284973 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:21.284943    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1beb1bd2-dcc6-4323-8c68-87c1960cb0b6" path="/var/lib/kubelet/pods/1beb1bd2-dcc6-4323-8c68-87c1960cb0b6/volumes"
Oct 30 05:52:21.285500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:21.285484    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8854e3da-6a3e-498d-b447-83ac72033214" path="/var/lib/kubelet/pods/8854e3da-6a3e-498d-b447-83ac72033214/volumes"
Oct 30 05:52:21.521426 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:21.521354    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:21.685082 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:21.685046    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:21.838872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:21.838781    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:22.033224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:22.027925    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:22.670250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:22.669592    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:23.465870 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:23.465842    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:23.609907 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:23.609863    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:24.271445 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:24.271410    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:24.482971 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:24.482933    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:25.262331 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:25.262290    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:25.585539 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:25.585500    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:25.924701 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:25.924672    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:26.083797 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:26.083763    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:27.383905 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:27.383872    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:27.537998 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:27.537962    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:27.709697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:27.709401    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:28.988004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:28.987956    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:29.139021 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:29.138982    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:29.822590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:29.822554    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:30.172581 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:30.172543    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:31.717594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:31.717554    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:32.331576 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:32.331524    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:32.373584 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:52:32.373552    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:52:33.156193 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:33.156156    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:34.783532 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:34.783502    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"]
Oct 30 05:52:34.784083 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:34.783723    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy" containerID="cri-o://45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d" gracePeriod=30
Oct 30 05:52:34.913062 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:34.913033    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:35.455931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:35.455900    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:36.699046 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:36.699014    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:36.869462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:36.869432    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:37.094037 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:37.093961    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:37.540437 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:37.540405    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:38.602034 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:38.602004    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:40.064541 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.064482    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:52:40.214552 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.214514    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-kube-api-access-z8dh6\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-kube-api-access-z8dh6\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.214725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.214647    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-podinfo\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.214725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.214697    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-envoy\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.214836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.214777    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-token\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.214836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.214828    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istiod-ca-cert\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.214951 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.214889    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-socket\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.215011 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215003    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-credential-socket\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.215071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215055    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-data\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.215126 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215112    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-certs\") pod \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\" (UID: \"6c4fc278-f251-4c6a-9c0d-1a185950d6ef\") "
Oct 30 05:52:40.215212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215147    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-socket" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:40.215662 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215583    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-credential-socket" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:40.215662 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215597    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-data" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:40.215845 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215821    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-certs" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:40.215909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215828    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.215909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215870    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.215909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.215888    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.216230 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.216208    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istiod-ca-cert" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:52:40.217330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.217298    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-podinfo" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:52:40.217330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.217324    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-envoy" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:52:40.217575 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.217559    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-token" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:40.217643 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.217625    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-kube-api-access-z8dh6" pod "6c4fc278-f251-4c6a-9c0d-1a185950d6ef" (UID: "6c4fc278-f251-4c6a-9c0d-1a185950d6ef"). InnerVolumeSpecName "kube-api-access-z8dh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:40.318105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.318010    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.318105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.318050    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.318105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.318065    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.318105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.318078    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.318105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.318090    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.318105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.318102    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dh6\" (UniqueName: \"kubernetes.io/projected/6c4fc278-f251-4c6a-9c0d-1a185950d6ef-kube-api-access-z8dh6\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:52:40.494107 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.494074    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:40.525267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.525219    5250 generic.go:356] "Generic (PLEG): container finished" podID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerID="45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d" exitCode=0
Oct 30 05:52:40.525434 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.525283    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"
Oct 30 05:52:40.525434 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.525290    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" event={"ID":"6c4fc278-f251-4c6a-9c0d-1a185950d6ef","Type":"ContainerDied","Data":"45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d"}
Oct 30 05:52:40.525434 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.525347    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95" event={"ID":"6c4fc278-f251-4c6a-9c0d-1a185950d6ef","Type":"ContainerDied","Data":"f0c88eff0ff6095d1d9c82d38bdd838c9d0474d592586d398ac91fff0807e4d4"}
Oct 30 05:52:40.525434 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.525370    5250 scope.go:122] "RemoveContainer" containerID="45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d"
Oct 30 05:52:40.536912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.536897    5250 scope.go:122] "RemoveContainer" containerID="45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d"
Oct 30 05:52:40.537146 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:52:40.537113    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d\": container with ID starting with 45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d not found: ID does not exist" containerID="45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d"
Oct 30 05:52:40.537231 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.537158    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d"} err="failed to get container status \"45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d\": rpc error: code = NotFound desc = could not find container \"45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d\": container with ID starting with 45b2f748f6ba95bca5d05e5076774afb896214ca624a8d57a611f6df2ff3015d not found: ID does not exist"
Oct 30 05:52:40.585215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.585120    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"]
Oct 30 05:52:40.598229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.598173    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-c6w95"]
Oct 30 05:52:40.658367 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.658278    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:40.924669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:40.924627    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:41.281927 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:41.281841    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" path="/var/lib/kubelet/pods/6c4fc278-f251-4c6a-9c0d-1a185950d6ef/volumes"
Oct 30 05:52:42.310205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:42.309794    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:42.966776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:42.966718    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:44.908006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:44.907975    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:46.428559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:46.428528    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:46.588370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:46.588330    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:48.264908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:48.264875    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:50.083962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:50.083919    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:51.595085 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:51.595048    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:53.098149 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:53.098050    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:54.765879 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:54.765845    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:56.252328 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:56.252274    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:57.829498 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:57.829466    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:59.764548 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:52:59.764510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:00.530400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:00.530363    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:00.977061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:00.977027    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:01.451046 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:01.450993    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:01.606326 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:01.606295    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:01.756782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:01.756667    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:01.897758 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:01.897724    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:02.287453 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.287423    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"]
Oct 30 05:53:02.288521 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288497    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy"
Oct 30 05:53:02.288521 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288520    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy"
Oct 30 05:53:02.288714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288540    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy"
Oct 30 05:53:02.288714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288549    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy"
Oct 30 05:53:02.288714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288564    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8854e3da-6a3e-498d-b447-83ac72033214" containerName="payload-processing"
Oct 30 05:53:02.288714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288572    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8854e3da-6a3e-498d-b447-83ac72033214" containerName="payload-processing"
Oct 30 05:53:02.288714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288589    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1beb1bd2-dcc6-4323-8c68-87c1960cb0b6" containerName="payload-pre-processing"
Oct 30 05:53:02.288714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288597    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1beb1bd2-dcc6-4323-8c68-87c1960cb0b6" containerName="payload-pre-processing"
Oct 30 05:53:02.289004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288784    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy"
Oct 30 05:53:02.289004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288803    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy"
Oct 30 05:53:02.289004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288815    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4fc278-f251-4c6a-9c0d-1a185950d6ef" containerName="istio-proxy"
Oct 30 05:53:02.289004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288825    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8854e3da-6a3e-498d-b447-83ac72033214" containerName="payload-processing"
Oct 30 05:53:02.289004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.288838    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1beb1bd2-dcc6-4323-8c68-87c1960cb0b6" containerName="payload-pre-processing"
Oct 30 05:53:02.292121 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.292100    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.339310 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.339283    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a1c17ebc-openshift-default-dockercfg-dsrn9\""
Oct 30 05:53:02.356319 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.356281    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"]
Oct 30 05:53:02.357411 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.357379    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-envoy\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.357519 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.357418    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaf27-0be7-44d2-8e5e-01986e050701-istio-podinfo\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.357625 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.357552    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-socket\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.357710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.357696    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-credential-socket\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.357768 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.357753    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-data\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.357873 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.357860    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-istio-token\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.357935 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.357909    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96n45\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-kube-api-access-96n45\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.358058 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.358045    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-certs\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.358117 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.358093    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaf27-0be7-44d2-8e5e-01986e050701-istiod-ca-cert\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.385405 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:53:02.385369    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:53:02.469894 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.469859    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-certs\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.470163 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.470140    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaf27-0be7-44d2-8e5e-01986e050701-istiod-ca-cert\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.489078 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.489046    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-envoy\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.489739 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.489713    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaf27-0be7-44d2-8e5e-01986e050701-istio-podinfo\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.489964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.489946    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-socket\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.491210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.470986    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaf27-0be7-44d2-8e5e-01986e050701-istiod-ca-cert\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.491210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.484765    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:02.491210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.471326    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-certs\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.491210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.490346    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-socket\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.491656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.491628    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-credential-socket\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.492127 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.492106    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-data\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.492396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.492374    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-istio-token\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.492554 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.492537    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96n45\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-kube-api-access-96n45\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.493022 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.492029    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-credential-socket\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.493425 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.493405    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-data\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.537308 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.530341    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-istio-token\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.538727 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.538151    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-envoy\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.561002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.560917    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaf27-0be7-44d2-8e5e-01986e050701-istio-podinfo\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.574213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.563501    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96n45\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-kube-api-access-96n45\") pod \"e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") " pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:02.609090 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:02.609058    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:03.016248 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:03.015226    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"]
Oct 30 05:53:03.016248 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:53:03.015833    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389eaf27_0be7_44d2_8e5e_01986e050701.slice/crio-84238c9c75c9ecfa7662ba0746cc144217a0dad131aa9639d38cba6eb31df235 WatchSource:0}: Error finding container 84238c9c75c9ecfa7662ba0746cc144217a0dad131aa9639d38cba6eb31df235: Status 404 returned error can't find the container with id 84238c9c75c9ecfa7662ba0746cc144217a0dad131aa9639d38cba6eb31df235
Oct 30 05:53:03.019784 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:03.019743    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:03.019917 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:03.019828    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:03.019917 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:03.019877    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:03.411092 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:03.411058    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:03.605283 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:03.605232    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:03.756807 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:03.756698    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" event={"ID":"389eaf27-0be7-44d2-8e5e-01986e050701","Type":"ContainerStarted","Data":"649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726"}
Oct 30 05:53:03.756807 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:03.756772    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" event={"ID":"389eaf27-0be7-44d2-8e5e-01986e050701","Type":"ContainerStarted","Data":"84238c9c75c9ecfa7662ba0746cc144217a0dad131aa9639d38cba6eb31df235"}
Oct 30 05:53:04.576317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.576273    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:04.611474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.611364    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:04.619430 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.619365    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:04.687934 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.687885    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" podStartSLOduration=2.687870275 podStartE2EDuration="2.687870275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:03.855392351 +0000 UTC m=+1298.686429568" watchObservedRunningTime="2026-10-30 05:53:04.687870275 +0000 UTC m=+1299.518907491"
Oct 30 05:53:04.766554 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.766520    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:04.767634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.767611    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:53:04.928854 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.928820    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"]
Oct 30 05:53:04.945207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.938672    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy"
Oct 30 05:53:04.945466 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.945435    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab10583-5b3b-4092-83e0-beee39ba10a7" containerName="istio-proxy"
Oct 30 05:53:04.950217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.950193    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:04.967767 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:04.967741    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"]
Oct 30 05:53:05.044475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.044441    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.044679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.044651    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.044775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.044758    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.044831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.044791    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.044831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.044812    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.044931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.044873    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.044931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.044912    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cc7\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-kube-api-access-j5cc7\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.045029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.044943    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.045087 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.045059    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146085 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146050    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146116    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146142    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146233    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146280    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146336    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cc7\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-kube-api-access-j5cc7\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146386    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146442    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146481    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146505    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146527    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146605    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.146912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.146880    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.147074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.147056    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.148800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.148772    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.149106 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.149086    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.171008 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.170981    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.171387 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.171351    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cc7\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-kube-api-access-j5cc7\") pod \"maas-default-gateway-openshift-default-84587756dc-pkd9j\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.275502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.275413    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:05.330755 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.330726    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:05.511497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.511449    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:05.612102 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.612008    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"]
Oct 30 05:53:05.639524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.638600    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:05.639524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.638701    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:05.639524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.638744    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:05.698213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.698141    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:05.779055 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.778957    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" event={"ID":"4aa3bcb2-8640-47d5-a423-fda32c2d77d9","Type":"ContainerStarted","Data":"5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447"}
Oct 30 05:53:05.779055 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.779022    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" event={"ID":"4aa3bcb2-8640-47d5-a423-fda32c2d77d9","Type":"ContainerStarted","Data":"58589c07a38317ec685fece36c70f69d56000774ad375a890a7210ee3defa8a1"}
Oct 30 05:53:05.828614 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:05.828550    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podStartSLOduration=1.828528016 podStartE2EDuration="1.828528016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:05.827578503 +0000 UTC m=+1300.658615720" watchObservedRunningTime="2026-10-30 05:53:05.828528016 +0000 UTC m=+1300.659565234"
Oct 30 05:53:06.176942 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:06.176904    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:06.275949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:06.275917    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:06.277248 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:06.277222    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-pkd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 05:53:06.277378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:06.277267    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 05:53:06.368749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:06.368695    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:06.548904 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:06.548823    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:06.907923 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:06.907890    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:07.218687 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:07.218593    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:07.280244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:07.277634    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-pkd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 05:53:07.280244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:07.277697    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 05:53:07.404443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:07.404405    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:07.645376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:07.645337    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:07.821152 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:07.821091    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:08.237740 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:08.237648    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:08.280334 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:08.280292    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-pkd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 05:53:08.280496 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:08.280371    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 05:53:09.280844 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:09.280812    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-pkd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 05:53:09.281510 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:09.281474    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 05:53:09.980646 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:09.978343    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:10.192897 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:10.192824    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:10.282650 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:10.282544    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-pkd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 05:53:10.282650 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:10.282607    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 05:53:11.224130 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:11.224095    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:11.307351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:11.299600    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-pkd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 05:53:11.307351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:11.299679    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 05:53:12.276203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:12.276157    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-pkd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 05:53:12.276463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:12.276430    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 05:53:12.839375 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:12.839336    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:13.276722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:13.276682    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-pkd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 05:53:13.277015 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:13.276984    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 05:53:13.306547 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:13.306504    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:14.237941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:14.237899    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:14.588152 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:14.588047    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:15.047966 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:15.047927    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:15.140265 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:15.140232    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:15.140709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:15.140688    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:15.141578 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:15.141560    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 05:53:15.245002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:15.244969    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:15.729667 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:15.729630    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:15.883150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:15.883110    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:16.363098 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:16.363070    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:16.549494 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:16.549454    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:16.738090 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:16.738062    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:17.743219 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:17.743135    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:17.899167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:17.899139    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:18.276911 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:18.276868    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:18.443647 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:18.443620    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:18.597160 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:18.597082    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:18.817462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:18.816019    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:20.352369 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.352338    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:20.402026 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.401999    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"]
Oct 30 05:53:20.408245 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.408217    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:20.427043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.427017    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-37bbcc\""
Oct 30 05:53:20.431947 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.431925    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"]
Oct 30 05:53:20.436684 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.436661    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:20.443875 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.443821    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"]
Oct 30 05:53:20.457282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.457255    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-37bbcc-dockercfg-8mw6d\""
Oct 30 05:53:20.461150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.461121    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"]
Oct 30 05:53:20.535137 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.535091    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:20.541729 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.541702    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dd459d-f6af-45f3-820d-f57ce3409e29-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg\" (UID: \"c2dd459d-f6af-45f3-820d-f57ce3409e29\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:20.542002 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.541984    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fp7t\" (UniqueName: \"kubernetes.io/projected/4fda6e76-0dd9-4775-99ad-9242db7d63cd-kube-api-access-4fp7t\") pod \"payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx\" (UID: \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:20.542095 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.542076    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5wp\" (UniqueName: \"kubernetes.io/projected/c2dd459d-f6af-45f3-820d-f57ce3409e29-kube-api-access-jv5wp\") pod \"payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg\" (UID: \"c2dd459d-f6af-45f3-820d-f57ce3409e29\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:20.542312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.542290    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda6e76-0dd9-4775-99ad-9242db7d63cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx\" (UID: \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:20.643634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.643601    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fp7t\" (UniqueName: \"kubernetes.io/projected/4fda6e76-0dd9-4775-99ad-9242db7d63cd-kube-api-access-4fp7t\") pod \"payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx\" (UID: \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:20.643817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.643690    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5wp\" (UniqueName: \"kubernetes.io/projected/c2dd459d-f6af-45f3-820d-f57ce3409e29-kube-api-access-jv5wp\") pod \"payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg\" (UID: \"c2dd459d-f6af-45f3-820d-f57ce3409e29\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:20.643817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.643804    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda6e76-0dd9-4775-99ad-9242db7d63cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx\" (UID: \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:20.643895 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.643845    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dd459d-f6af-45f3-820d-f57ce3409e29-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg\" (UID: \"c2dd459d-f6af-45f3-820d-f57ce3409e29\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:20.644636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.644616    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dd459d-f6af-45f3-820d-f57ce3409e29-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg\" (UID: \"c2dd459d-f6af-45f3-820d-f57ce3409e29\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:20.644776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.644754    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda6e76-0dd9-4775-99ad-9242db7d63cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx\" (UID: \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:20.671223 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.670759    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5wp\" (UniqueName: \"kubernetes.io/projected/c2dd459d-f6af-45f3-820d-f57ce3409e29-kube-api-access-jv5wp\") pod \"payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg\" (UID: \"c2dd459d-f6af-45f3-820d-f57ce3409e29\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:20.671223 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.670830    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fp7t\" (UniqueName: \"kubernetes.io/projected/4fda6e76-0dd9-4775-99ad-9242db7d63cd-kube-api-access-4fp7t\") pod \"payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx\" (UID: \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:20.725444 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.725404    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:20.753604 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:20.752017    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:21.060631 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:21.060473    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"]
Oct 30 05:53:21.060914 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:53:21.060889    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fda6e76_0dd9_4775_99ad_9242db7d63cd.slice/crio-571c9286b0e4abc19b39474315965fcdb0e382845144057b4814d2083c1e155d WatchSource:0}: Error finding container 571c9286b0e4abc19b39474315965fcdb0e382845144057b4814d2083c1e155d: Status 404 returned error can't find the container with id 571c9286b0e4abc19b39474315965fcdb0e382845144057b4814d2083c1e155d
Oct 30 05:53:21.116869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:21.115445    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"]
Oct 30 05:53:21.978862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:21.978820    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx" event={"ID":"4fda6e76-0dd9-4775-99ad-9242db7d63cd","Type":"ContainerStarted","Data":"a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a"}
Oct 30 05:53:21.979332 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:21.978880    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx" event={"ID":"4fda6e76-0dd9-4775-99ad-9242db7d63cd","Type":"ContainerStarted","Data":"571c9286b0e4abc19b39474315965fcdb0e382845144057b4814d2083c1e155d"}
Oct 30 05:53:21.979332 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:21.978905    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:21.980681 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:21.980656    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg" event={"ID":"c2dd459d-f6af-45f3-820d-f57ce3409e29","Type":"ContainerStarted","Data":"fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c"}
Oct 30 05:53:21.980798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:21.980704    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg" event={"ID":"c2dd459d-f6af-45f3-820d-f57ce3409e29","Type":"ContainerStarted","Data":"890d6ff9b24619b1dd95684a564af8becd2cbc48f73987588c9ff578b3f78cf9"}
Oct 30 05:53:21.980798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:21.980743    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:22.022250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:22.022160    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx" podStartSLOduration=2.022142806 podStartE2EDuration="2.022142806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:22.021755835 +0000 UTC m=+1316.852793046" watchObservedRunningTime="2026-10-30 05:53:22.022142806 +0000 UTC m=+1316.853180025"
Oct 30 05:53:22.066425 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:22.066377    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg" podStartSLOduration=2.066361924 podStartE2EDuration="2.066361924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:22.063885312 +0000 UTC m=+1316.894922554" watchObservedRunningTime="2026-10-30 05:53:22.066361924 +0000 UTC m=+1316.897399140"
Oct 30 05:53:22.469097 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:22.469064    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:24.272564 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:24.272530    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:25.798057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:25.798003    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:27.384668 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:27.384635    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:28.035903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:28.035873    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"]
Oct 30 05:53:28.049286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:28.049261    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b15d0f-fsk2p"]
Oct 30 05:53:28.867415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:28.867383    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:29.289599 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:29.289566    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0aedd7-d585-4784-b9b5-0d9c9512339c" path="/var/lib/kubelet/pods/2d0aedd7-d585-4784-b9b5-0d9c9512339c/volumes"
Oct 30 05:53:30.009293 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.009258    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:30.614756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.614706    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:30.762381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.762347    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"]
Oct 30 05:53:30.768892 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.768869    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.800990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.800965    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-0b481b-openshift-default-dockercfg-nrg7r\""
Oct 30 05:53:30.805221 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.805191    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.805503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.805474    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.805638 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.805573    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f449w\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-kube-api-access-f449w\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.805710 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.805675    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.805896 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.805867    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.805981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.805914    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c88e80-0349-4523-bacc-048e69785ab9-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.806032 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.805976    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.806074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.806061    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.806216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.806195    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c88e80-0349-4523-bacc-048e69785ab9-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.807167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.807144    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"]
Oct 30 05:53:30.907727 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.907696    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f449w\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-kube-api-access-f449w\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.907943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.907811    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.907943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.907913    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.907952    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c88e80-0349-4523-bacc-048e69785ab9-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908004    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908031    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908323 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908166    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c88e80-0349-4523-bacc-048e69785ab9-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908323 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908220    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908323 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908256    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908487 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908329    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908487 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908433    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908605 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908479    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908665 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908644    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.908859 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.908840    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c88e80-0349-4523-bacc-048e69785ab9-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.910997 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.910969    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.911204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.911171    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c88e80-0349-4523-bacc-048e69785ab9-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.932761 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.932737    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:30.932889 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:30.932846    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f449w\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-kube-api-access-f449w\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:31.083481 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:31.083446    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:31.428595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:31.428563    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"]
Oct 30 05:53:31.429124 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:53:31.429096    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c88e80_0349_4523_bacc_048e69785ab9.slice/crio-d71d739da3ea5985505f9df6698ac643549dc2c767c8cd9bf35686c749b05f4a WatchSource:0}: Error finding container d71d739da3ea5985505f9df6698ac643549dc2c767c8cd9bf35686c749b05f4a: Status 404 returned error can't find the container with id d71d739da3ea5985505f9df6698ac643549dc2c767c8cd9bf35686c749b05f4a
Oct 30 05:53:31.431747 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:31.431716    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:31.431814 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:31.431785    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:31.431851 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:31.431812    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:32.058725 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.058689    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" event={"ID":"f9c88e80-0349-4523-bacc-048e69785ab9","Type":"ContainerStarted","Data":"208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd"}
Oct 30 05:53:32.058902 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.058742    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" event={"ID":"f9c88e80-0349-4523-bacc-048e69785ab9","Type":"ContainerStarted","Data":"d71d739da3ea5985505f9df6698ac643549dc2c767c8cd9bf35686c749b05f4a"}
Oct 30 05:53:32.084072 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.084042    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:32.085771 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.085746    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 05:53:32.085886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.085795    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 05:53:32.166608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.160581    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podStartSLOduration=2.160545735 podStartE2EDuration="2.160545735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:32.143032871 +0000 UTC m=+1326.974070090" watchObservedRunningTime="2026-10-30 05:53:32.160545735 +0000 UTC m=+1326.991582954"
Oct 30 05:53:32.397593 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:53:32.397561    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:53:32.545841 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.545777    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:32.690070 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.689978    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:32.897225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.897197    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:32.994591 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.994505    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 05:53:32.994855 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:32.994832    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 05:53:33.089297 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.086642    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 05:53:33.089297 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.086700    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 05:53:33.811600 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.811550    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"]
Oct 30 05:53:33.818196 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.818149    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.840591 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.840558    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"]
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855218    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855306    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855353    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855431    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87b4d\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-kube-api-access-87b4d\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855483    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855513    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855618    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855640    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.859201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.855694    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.960205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.960147    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87b4d\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-kube-api-access-87b4d\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.979579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.979549    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.980282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.980261    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.980919 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.980900    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.981376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.981358    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.981858 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.981838    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.982440 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.982420    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.983140 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.983097    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.990201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.987684    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.991330 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.981302    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.991501 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.982248    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.991596 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.987594    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.991673 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.991118    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.991746 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.980200    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.993705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.993675    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.993778 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.993719    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:33.999570 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:33.999548    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87b4d\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-kube-api-access-87b4d\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:34.006541 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.005951    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:34.093714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.092630    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 05:53:34.093714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.092694    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 05:53:34.148886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.148851    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:34.511604 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.511577    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"]
Oct 30 05:53:34.512471 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:53:34.512446    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ae5ad1_1ee9_4a65_a10f_72004c9ea214.slice/crio-a3c2d35cdefeb71a4fa6a1cbf2ac8b6ac838a9869fcd20455a0930aad656c7b7 WatchSource:0}: Error finding container a3c2d35cdefeb71a4fa6a1cbf2ac8b6ac838a9869fcd20455a0930aad656c7b7: Status 404 returned error can't find the container with id a3c2d35cdefeb71a4fa6a1cbf2ac8b6ac838a9869fcd20455a0930aad656c7b7
Oct 30 05:53:34.515647 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.515614    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:34.515772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.515695    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:34.515772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.515744    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:53:34.650198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.650156    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"]
Oct 30 05:53:34.654634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.654610    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.694194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.680556    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a1c17ebc-serving-cert\""
Oct 30 05:53:34.694194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.693000    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"]
Oct 30 05:53:34.787714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.787621    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:34.818826 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.818788    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w79c\" (UniqueName: \"kubernetes.io/projected/beb1ac85-0b11-48aa-adec-c2b4498ef693-kube-api-access-4w79c\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.819012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.818950    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-metrics-certs\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.819125 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.819109    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.819214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.819135    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1ac85-0b11-48aa-adec-c2b4498ef693-openshift-service-ca\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.920311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.920266    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.920518 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.920327    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1ac85-0b11-48aa-adec-c2b4498ef693-openshift-service-ca\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.920518 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.920413    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w79c\" (UniqueName: \"kubernetes.io/projected/beb1ac85-0b11-48aa-adec-c2b4498ef693-kube-api-access-4w79c\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.920518 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:53:34.920453    5250 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-serving-cert: secret "maas-api-e2e-mt-a1c17ebc-serving-cert" not found
Oct 30 05:53:34.920691 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:53:34.920533    5250 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls podName:beb1ac85-0b11-48aa-adec-c2b4498ef693 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:35.420509684 +0000 UTC m=+1330.251546882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls") pod "maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm" (UID: "beb1ac85-0b11-48aa-adec-c2b4498ef693") : secret "maas-api-e2e-mt-a1c17ebc-serving-cert" not found
Oct 30 05:53:34.920691 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.920606    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-metrics-certs\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.921441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.921406    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1ac85-0b11-48aa-adec-c2b4498ef693-openshift-service-ca\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.923305 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.923286    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-metrics-certs\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:34.932457 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.932433    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:34.946116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:34.946092    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w79c\" (UniqueName: \"kubernetes.io/projected/beb1ac85-0b11-48aa-adec-c2b4498ef693-kube-api-access-4w79c\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:35.093697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.093610    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 05:53:35.093697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.093673    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 05:53:35.142705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.142669    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" event={"ID":"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214","Type":"ContainerStarted","Data":"baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a"}
Oct 30 05:53:35.142879 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.142736    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" event={"ID":"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214","Type":"ContainerStarted","Data":"a3c2d35cdefeb71a4fa6a1cbf2ac8b6ac838a9869fcd20455a0930aad656c7b7"}
Oct 30 05:53:35.149941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.149913    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:35.151417 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.151393    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:53:35.151544 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.151436    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:53:35.185459 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.185427    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:35.217877 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.217821    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podStartSLOduration=2.217804631 podStartE2EDuration="2.217804631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:35.212891281 +0000 UTC m=+1330.043928499" watchObservedRunningTime="2026-10-30 05:53:35.217804631 +0000 UTC m=+1330.048841919"
Oct 30 05:53:35.431004 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.430962    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:35.433731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.433703    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:35.620200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:35.620024    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:36.025447 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:36.025417    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"]
Oct 30 05:53:36.105205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:36.091085    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 05:53:36.105205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:36.091138    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 05:53:36.150928 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:36.150091    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:53:36.150928 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:36.150155    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:53:36.163203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:36.155210    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm" event={"ID":"beb1ac85-0b11-48aa-adec-c2b4498ef693","Type":"ContainerStarted","Data":"f221e2422748b8c2881fa796b6bc555966cc84b878231dc99f799d1ab5e96dd4"}
Oct 30 05:53:37.097652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.095292    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 05:53:37.097652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.095360    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 05:53:37.149924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.149888    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:53:37.150762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.149958    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:53:37.216984 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.216944    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm" event={"ID":"beb1ac85-0b11-48aa-adec-c2b4498ef693","Type":"ContainerStarted","Data":"436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6"}
Oct 30 05:53:37.217398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.217378    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:37.316959 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.316496    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm" podStartSLOduration=2.911543918 podStartE2EDuration="3.316472934s" totalImagesPullingTime="404.929016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:37.27064353 +0000 UTC m=+1332.101680749" watchObservedRunningTime="2026-10-30 05:53:37.316472934 +0000 UTC m=+1332.147510151"
Oct 30 05:53:37.645160 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.645130    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:37.927887 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:37.927728    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:38.099194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:38.099146    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:38.099849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:38.099820    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:38.100840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:38.100821    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 05:53:38.149925 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:38.149893    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:53:38.150442 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:38.149950    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:53:39.149741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:39.149712    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:53:39.149953 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:39.149768    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:53:40.149280 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:40.149247    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:53:40.149454 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:40.149309    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:53:40.175712 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:40.175674    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:41.150257 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:41.150219    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:53:41.150566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:41.150536    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:53:42.150343 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:42.150313    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:53:42.150763 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:42.150363    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:53:43.247197 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:43.245280    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:53:43.603803 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:43.603666    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:43.612150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:43.612120    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:43.627897 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:43.627845    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 05:53:43.669270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:43.669220    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 05:53:43.729463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:43.722972    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:45.030524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:45.030494    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"]
Oct 30 05:53:45.043395 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:45.043371    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3647e369-7rz65"]
Oct 30 05:53:45.281968 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:45.281892    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d" path="/var/lib/kubelet/pods/46fe9d4f-6b5b-4bb8-9eff-d8cf6a2ad15d/volumes"
Oct 30 05:53:45.706355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:45.706226    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:47.555783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:47.555751    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:47.714081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:47.714050    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:49.381867 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:49.381806    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:50.889214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:50.886260    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:52.741723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:52.741644    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:54.287362 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:54.287330    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:55.866964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:55.866929    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:57.433404 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:57.433365    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:58.040853 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:58.040767    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"]
Oct 30 05:53:58.052194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:58.052152    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916f3685-mpwxd"]
Oct 30 05:53:58.850916 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:58.850858    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:59.282366 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:59.282336    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4e35f9-3cda-4b70-a615-197aeb09fb32" path="/var/lib/kubelet/pods/7e4e35f9-3cda-4b70-a615-197aeb09fb32/volumes"
Oct 30 05:53:59.294752 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:53:59.294728    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:00.375139 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:00.375105    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:02.257898 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:02.257860    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:02.388612 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:02.388508    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:54:04.216636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:04.216594    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:05.906107 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:05.906018    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:06.149168 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:06.149131    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:06.446830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:06.446797    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:07.293432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.293397    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8874-7744ec"]
Oct 30 05:54:07.298998 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.298976    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-7744ec"
Oct 30 05:54:07.314503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.314474    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-7744ec"]
Oct 30 05:54:07.316563 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.316537    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bnq\""
Oct 30 05:54:07.351168 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.351112    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxf4\" (UniqueName: \"kubernetes.io/projected/5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b-kube-api-access-wqxf4\") pod \"test-curl-8874-7744ec\" (UID: \"5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b\") " pod="opendatahub/test-curl-8874-7744ec"
Oct 30 05:54:07.412389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.412354    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:07.452748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.452710    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxf4\" (UniqueName: \"kubernetes.io/projected/5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b-kube-api-access-wqxf4\") pod \"test-curl-8874-7744ec\" (UID: \"5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b\") " pod="opendatahub/test-curl-8874-7744ec"
Oct 30 05:54:07.478220 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.478166    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxf4\" (UniqueName: \"kubernetes.io/projected/5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b-kube-api-access-wqxf4\") pod \"test-curl-8874-7744ec\" (UID: \"5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b\") " pod="opendatahub/test-curl-8874-7744ec"
Oct 30 05:54:07.556488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.556403    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:07.615840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.615798    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-7744ec"
Oct 30 05:54:07.930496 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:07.930470    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-7744ec"]
Oct 30 05:54:07.931113 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:07.931089    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c78ffea_eb1e_4336_bf64_0ce17f7d0e9b.slice/crio-b68f87c3adb4430fb61048e825c369aa3382ef8509ccdbf83a2c536b43682ba4 WatchSource:0}: Error finding container b68f87c3adb4430fb61048e825c369aa3382ef8509ccdbf83a2c536b43682ba4: Status 404 returned error can't find the container with id b68f87c3adb4430fb61048e825c369aa3382ef8509ccdbf83a2c536b43682ba4
Oct 30 05:54:08.580584 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:08.580543    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-7744ec" event={"ID":"5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b","Type":"ContainerStarted","Data":"15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304"}
Oct 30 05:54:08.580985 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:08.580592    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-7744ec" event={"ID":"5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b","Type":"ContainerStarted","Data":"b68f87c3adb4430fb61048e825c369aa3382ef8509ccdbf83a2c536b43682ba4"}
Oct 30 05:54:09.013074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.013021    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8874-7744ec" podStartSLOduration=2.013004675 podStartE2EDuration="2.013004675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:08.635088505 +0000 UTC m=+1363.466125722" watchObservedRunningTime="2026-10-30 05:54:09.013004675 +0000 UTC m=+1363.844041894"
Oct 30 05:54:09.017544 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.017516    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"]
Oct 30 05:54:09.022055 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.022036    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.046501 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.045368    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"]
Oct 30 05:54:09.079661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.079618    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-dshm\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.079856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.079700    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-model-cache\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.079856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.079776    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krm25\" (UniqueName: \"kubernetes.io/projected/1a6dbe69-607b-4640-bf3a-d5139a746e1c-kube-api-access-krm25\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.080038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.079998    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tls-certs\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.080248 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.080225    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-home\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.080352 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.080282    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tmp-dir\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.092238 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.092191    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:09.131481 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.131444    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a1c17ebc\"/\"openshift-service-ca.crt\""
Oct 30 05:54:09.131644 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.131576    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a1c17ebc\"/\"e2e-lifecycle-model-a1c17ebc-kserve-self-signed-certs\""
Oct 30 05:54:09.131644 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.131580    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a1c17ebc\"/\"kube-root-ca.crt\""
Oct 30 05:54:09.131644 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.131639    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a1c17ebc\"/\"default-dockercfg-vql6c\""
Oct 30 05:54:09.181571 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.181538    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-dshm\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.181816 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.181605    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-model-cache\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.181816 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.181684    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krm25\" (UniqueName: \"kubernetes.io/projected/1a6dbe69-607b-4640-bf3a-d5139a746e1c-kube-api-access-krm25\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.181945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.181902    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tls-certs\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.181945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.181934    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-home\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.182060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.181997    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tmp-dir\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.182112 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.182065    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-model-cache\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.182303 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.182270    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-home\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.182544 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.182400    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tmp-dir\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.184116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.184089    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-dshm\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.185036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.185013    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tls-certs\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.208601 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.208573    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krm25\" (UniqueName: \"kubernetes.io/projected/1a6dbe69-607b-4640-bf3a-d5139a746e1c-kube-api-access-krm25\") pod \"e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") " pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.282021 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.281913    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:09.344257 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.341719    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:09.548474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.543618    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8874-7744ec"]
Oct 30 05:54:09.548474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.544447    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8874-7744ec"]
Oct 30 05:54:09.698939 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.698907    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8874-1ced51"]
Oct 30 05:54:09.700019 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.699992    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b" containerName="test-curl-8874-7744ec"
Oct 30 05:54:09.700141 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.700021    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b" containerName="test-curl-8874-7744ec"
Oct 30 05:54:09.700265 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.700217    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b" containerName="test-curl-8874-7744ec"
Oct 30 05:54:09.709278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.709234    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-1ced51"
Oct 30 05:54:09.719312 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.719276    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-1ced51"]
Oct 30 05:54:09.734806 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.734774    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:09.773227 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:09.772609    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a6dbe69_607b_4640_bf3a_d5139a746e1c.slice/crio-3f15ceaa86b3448b2b57aaf43ef93741c67b5c866b1a2ff95646bb2a37a4307a WatchSource:0}: Error finding container 3f15ceaa86b3448b2b57aaf43ef93741c67b5c866b1a2ff95646bb2a37a4307a: Status 404 returned error can't find the container with id 3f15ceaa86b3448b2b57aaf43ef93741c67b5c866b1a2ff95646bb2a37a4307a
Oct 30 05:54:09.777463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.777417    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"]
Oct 30 05:54:09.811421 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.811345    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55vw\" (UniqueName: \"kubernetes.io/projected/ddf92670-9e98-4e8c-bed5-6557f8a136b0-kube-api-access-n55vw\") pod \"test-curl-8874-1ced51\" (UID: \"ddf92670-9e98-4e8c-bed5-6557f8a136b0\") " pod="opendatahub/test-curl-8874-1ced51"
Oct 30 05:54:09.912823 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.912789    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55vw\" (UniqueName: \"kubernetes.io/projected/ddf92670-9e98-4e8c-bed5-6557f8a136b0-kube-api-access-n55vw\") pod \"test-curl-8874-1ced51\" (UID: \"ddf92670-9e98-4e8c-bed5-6557f8a136b0\") " pod="opendatahub/test-curl-8874-1ced51"
Oct 30 05:54:09.937515 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:09.937479    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55vw\" (UniqueName: \"kubernetes.io/projected/ddf92670-9e98-4e8c-bed5-6557f8a136b0-kube-api-access-n55vw\") pod \"test-curl-8874-1ced51\" (UID: \"ddf92670-9e98-4e8c-bed5-6557f8a136b0\") " pod="opendatahub/test-curl-8874-1ced51"
Oct 30 05:54:10.036574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:10.036370    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-1ced51"
Oct 30 05:54:10.317202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:10.317017    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:10.412208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:10.394705    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-1ced51"]
Oct 30 05:54:10.534235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:10.522470    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:10.654270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:10.653911    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" event={"ID":"1a6dbe69-607b-4640-bf3a-d5139a746e1c","Type":"ContainerStarted","Data":"98ade034837cfdd3ac80dc43cf8d333c2e32b7bdb1aeb1578f1633f0fac1d461"}
Oct 30 05:54:10.655331 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:10.655295    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" event={"ID":"1a6dbe69-607b-4640-bf3a-d5139a746e1c","Type":"ContainerStarted","Data":"3f15ceaa86b3448b2b57aaf43ef93741c67b5c866b1a2ff95646bb2a37a4307a"}
Oct 30 05:54:10.657286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:10.657258    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-1ced51" event={"ID":"ddf92670-9e98-4e8c-bed5-6557f8a136b0","Type":"ContainerStarted","Data":"0fe92067c9c0b8d4f628ed3c316464f060d7d0d0f83c644e029cd168d49dbb30"}
Oct 30 05:54:10.657642 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:10.657527    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8874-7744ec" podUID="5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b" containerName="test-curl-8874-7744ec" containerID="cri-o://15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304" gracePeriod=2
Oct 30 05:54:10.722455 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:10.722416    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-7744ec\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b" pod="opendatahub/test-curl-8874-7744ec"
Oct 30 05:54:11.300062 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:11.300028    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:11.343135 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:11.343101    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:11.667148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:11.667111    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-1ced51" event={"ID":"ddf92670-9e98-4e8c-bed5-6557f8a136b0","Type":"ContainerStarted","Data":"5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35"}
Oct 30 05:54:11.708382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:11.708328    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" podStartSLOduration=3.708314547 podStartE2EDuration="3.708314547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:10.707734636 +0000 UTC m=+1365.538771854" watchObservedRunningTime="2026-10-30 05:54:11.708314547 +0000 UTC m=+1366.539351778"
Oct 30 05:54:11.710497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:11.710454    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8874-1ced51" podStartSLOduration=2.7104414500000003 podStartE2EDuration="2.71044145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:11.709343103 +0000 UTC m=+1366.540380322" watchObservedRunningTime="2026-10-30 05:54:11.71044145 +0000 UTC m=+1366.541478667"
Oct 30 05:54:12.691611 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:12.691582    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8874-1ced51"]
Oct 30 05:54:12.698545 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:12.698519    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8874-1ced51"]
Oct 30 05:54:12.862359 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:12.862325    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8874-b43932"]
Oct 30 05:54:12.863502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:12.863474    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ddf92670-9e98-4e8c-bed5-6557f8a136b0" containerName="test-curl-8874-1ced51"
Oct 30 05:54:12.863666 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:12.863652    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf92670-9e98-4e8c-bed5-6557f8a136b0" containerName="test-curl-8874-1ced51"
Oct 30 05:54:12.866116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:12.863991    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf92670-9e98-4e8c-bed5-6557f8a136b0" containerName="test-curl-8874-1ced51"
Oct 30 05:54:12.978392 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:12.978317    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-b43932"]
Oct 30 05:54:12.978699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:12.978686    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:13.071118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.071090    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-7744ec"
Oct 30 05:54:13.100959 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.100921    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ncz\" (UniqueName: \"kubernetes.io/projected/a5b67951-a4ed-4e76-822e-07b6b740bbed-kube-api-access-75ncz\") pod \"test-curl-8874-b43932\" (UID: \"a5b67951-a4ed-4e76-822e-07b6b740bbed\") " pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:13.202450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.202413    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b-kube-api-access-wqxf4\" (UniqueName: \"kubernetes.io/projected/5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b-kube-api-access-wqxf4\") pod \"5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b\" (UID: \"5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b\") "
Oct 30 05:54:13.203225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.203163    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75ncz\" (UniqueName: \"kubernetes.io/projected/a5b67951-a4ed-4e76-822e-07b6b740bbed-kube-api-access-75ncz\") pod \"test-curl-8874-b43932\" (UID: \"a5b67951-a4ed-4e76-822e-07b6b740bbed\") " pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:13.204846 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.204823    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b-kube-api-access-wqxf4" pod "5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b" (UID: "5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b"). InnerVolumeSpecName "kube-api-access-wqxf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:54:13.228559 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.228494    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ncz\" (UniqueName: \"kubernetes.io/projected/a5b67951-a4ed-4e76-822e-07b6b740bbed-kube-api-access-75ncz\") pod \"test-curl-8874-b43932\" (UID: \"a5b67951-a4ed-4e76-822e-07b6b740bbed\") " pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:13.284041 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.284002    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b" path="/var/lib/kubelet/pods/5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b/volumes"
Oct 30 05:54:13.304766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.304735    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxf4\" (UniqueName: \"kubernetes.io/projected/5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b-kube-api-access-wqxf4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:54:13.308063 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.308038    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:13.496158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.496077    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:13.628426 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.628397    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-b43932"]
Oct 30 05:54:13.629544 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:13.629518    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5b67951_a4ed_4e76_822e_07b6b740bbed.slice/crio-68f13eabf544a2135615221ab2c211c63b4a8feb3ead0901c71d15bfca7ac31d WatchSource:0}: Error finding container 68f13eabf544a2135615221ab2c211c63b4a8feb3ead0901c71d15bfca7ac31d: Status 404 returned error can't find the container with id 68f13eabf544a2135615221ab2c211c63b4a8feb3ead0901c71d15bfca7ac31d
Oct 30 05:54:13.686248 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.686219    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-b43932" event={"ID":"a5b67951-a4ed-4e76-822e-07b6b740bbed","Type":"ContainerStarted","Data":"68f13eabf544a2135615221ab2c211c63b4a8feb3ead0901c71d15bfca7ac31d"}
Oct 30 05:54:13.688276 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.688246    5250 generic.go:356] "Generic (PLEG): container finished" podID="5c78ffea-eb1e-4336-bf64-0ce17f7d0e9b" containerID="15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304" exitCode=137
Oct 30 05:54:13.688428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.688318    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-7744ec"
Oct 30 05:54:13.688428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.688373    5250 scope.go:122] "RemoveContainer" containerID="15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304"
Oct 30 05:54:13.688428 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.688405    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8874-1ced51" podUID="ddf92670-9e98-4e8c-bed5-6557f8a136b0" containerName="test-curl-8874-1ced51" containerID="cri-o://5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35" gracePeriod=2
Oct 30 05:54:13.703599 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.703559    5250 scope.go:122] "RemoveContainer" containerID="15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304"
Oct 30 05:54:13.703916 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:13.703874    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304\": container with ID starting with 15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304 not found: ID does not exist" containerID="15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304"
Oct 30 05:54:13.703967 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:13.703926    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304"} err="failed to get container status \"15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304\": rpc error: code = NotFound desc = could not find container \"15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304\": container with ID starting with 15cb0a390526caca9b8a357948afa3182fd6132a994f9d4d2e4bd5b1a4d6a304 not found: ID does not exist"
Oct 30 05:54:14.040560 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:14.040520    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:14.697835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:14.697799    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-b43932" event={"ID":"a5b67951-a4ed-4e76-822e-07b6b740bbed","Type":"ContainerStarted","Data":"5878625800ae1b0616e774a44d5f3b5dd80f371cef0b7a2d3e81631a7c8e9d41"}
Oct 30 05:54:14.729827 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:14.729798    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:14.741517 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:14.741466    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8874-b43932" podStartSLOduration=2.74145299 podStartE2EDuration="2.74145299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:14.740487665 +0000 UTC m=+1369.571524885" watchObservedRunningTime="2026-10-30 05:54:14.74145299 +0000 UTC m=+1369.572490206"
Oct 30 05:54:14.919536 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:14.919500    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:15.602474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.602446    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:15.768372 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.768346    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8874-b43932"]
Oct 30 05:54:15.773152 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.773130    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8874-b43932"]
Oct 30 05:54:15.809414 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.808026    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:15.905950 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.905922    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8874-3948c0"]
Oct 30 05:54:15.906633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.906618    5250 cpu_manager.go:414] "containerMap: removing container" podUID="a5b67951-a4ed-4e76-822e-07b6b740bbed" containerName="test-curl-8874-b43932"
Oct 30 05:54:15.906633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.906634    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b67951-a4ed-4e76-822e-07b6b740bbed" containerName="test-curl-8874-b43932"
Oct 30 05:54:15.906740 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.906732    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b67951-a4ed-4e76-822e-07b6b740bbed" containerName="test-curl-8874-b43932"
Oct 30 05:54:15.910315 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.910298    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:15.932697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.932573    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-3948c0"]
Oct 30 05:54:15.959937 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.959912    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-1ced51"
Oct 30 05:54:15.980985 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.980956    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"]
Oct 30 05:54:15.988012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:15.987990    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.014068 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.014038    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"]
Oct 30 05:54:16.065775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.065740    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf92670-9e98-4e8c-bed5-6557f8a136b0-kube-api-access-n55vw\" (UniqueName: \"kubernetes.io/projected/ddf92670-9e98-4e8c-bed5-6557f8a136b0-kube-api-access-n55vw\") pod \"ddf92670-9e98-4e8c-bed5-6557f8a136b0\" (UID: \"ddf92670-9e98-4e8c-bed5-6557f8a136b0\") "
Oct 30 05:54:16.066113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066081    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.066289 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066148    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.066289 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066199    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.066289 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066266    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.066472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066358    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.066472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066419    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.066472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066452    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.067001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066897    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrrh\" (UniqueName: \"kubernetes.io/projected/ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5-kube-api-access-rhrrh\") pod \"test-curl-8874-3948c0\" (UID: \"ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5\") " pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:16.067001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.066983    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4djs\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-kube-api-access-f4djs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.067261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.067040    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.086253 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.086228    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf92670-9e98-4e8c-bed5-6557f8a136b0-kube-api-access-n55vw" pod "ddf92670-9e98-4e8c-bed5-6557f8a136b0" (UID: "ddf92670-9e98-4e8c-bed5-6557f8a136b0"). InnerVolumeSpecName "kube-api-access-n55vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.178655    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.179501    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.179727    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.179768    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.179830    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.179916    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.180036    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.180111    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.180144    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.180313    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrrh\" (UniqueName: \"kubernetes.io/projected/ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5-kube-api-access-rhrrh\") pod \"test-curl-8874-3948c0\" (UID: \"ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5\") " pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.180396    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4djs\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-kube-api-access-f4djs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.180473    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.180633    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n55vw\" (UniqueName: \"kubernetes.io/projected/ddf92670-9e98-4e8c-bed5-6557f8a136b0-kube-api-access-n55vw\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.181536    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.182311    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.182981    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.191252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.189405    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.222204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.219700    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrrh\" (UniqueName: \"kubernetes.io/projected/ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5-kube-api-access-rhrrh\") pod \"test-curl-8874-3948c0\" (UID: \"ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5\") " pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:16.228208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.227823    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.235203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.229428    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.235203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.231172    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4djs\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-kube-api-access-f4djs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.258203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.257973    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:16.305764 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.305733    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:16.679663 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.679635    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8874-3948c0"]
Oct 30 05:54:16.680212 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:16.680169    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72 WatchSource:0}: Error finding container 10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72: Status 404 returned error can't find the container with id 10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72
Oct 30 05:54:16.733239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.732910    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:16.733239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.732975    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:16.733239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.733003    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:16.739430 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.738094    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"]
Oct 30 05:54:16.779129 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.779094    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-3948c0" event={"ID":"ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5","Type":"ContainerStarted","Data":"10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72"}
Oct 30 05:54:16.780628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.780599    5250 generic.go:356] "Generic (PLEG): container finished" podID="ddf92670-9e98-4e8c-bed5-6557f8a136b0" containerID="5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35" exitCode=137
Oct 30 05:54:16.780765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.780642    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-1ced51"
Oct 30 05:54:16.780765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.780710    5250 scope.go:122] "RemoveContainer" containerID="5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35"
Oct 30 05:54:16.782659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.782625    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8874-b43932" podUID="a5b67951-a4ed-4e76-822e-07b6b740bbed" containerName="test-curl-8874-b43932" containerID="cri-o://5878625800ae1b0616e774a44d5f3b5dd80f371cef0b7a2d3e81631a7c8e9d41" gracePeriod=2
Oct 30 05:54:16.783029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.783001    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" event={"ID":"12fa15c4-7aa4-4664-8a37-10a6f3f2e214","Type":"ContainerStarted","Data":"e28551dcb2a8ee6e9b041c514a42e72a4a08cf3a0afc71b369e3f532f1f80aa2"}
Oct 30 05:54:16.809259 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:16.800046    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-b43932\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="a5b67951-a4ed-4e76-822e-07b6b740bbed" pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:16.809259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.805037    5250 scope.go:122] "RemoveContainer" containerID="5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35"
Oct 30 05:54:16.809259 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:16.808889    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35\": container with ID starting with 5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35 not found: ID does not exist" containerID="5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35"
Oct 30 05:54:16.809259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.808921    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35"} err="failed to get container status \"5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35\": rpc error: code = NotFound desc = could not find container \"5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35\": container with ID starting with 5e939d06ee62394268f8960ab0e2c7b61779fbc4b7cf1087d39070fba26bbe35 not found: ID does not exist"
Oct 30 05:54:16.815485 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:16.815447    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-b43932\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="a5b67951-a4ed-4e76-822e-07b6b740bbed" pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:16.836866 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:16.836410    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-1ced51\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="ddf92670-9e98-4e8c-bed5-6557f8a136b0" pod="opendatahub/test-curl-8874-1ced51"
Oct 30 05:54:16.938250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:16.923683    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:17.286580 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.286475    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf92670-9e98-4e8c-bed5-6557f8a136b0" path="/var/lib/kubelet/pods/ddf92670-9e98-4e8c-bed5-6557f8a136b0/volumes"
Oct 30 05:54:17.306946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.306916    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:17.726832 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.726797    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:17.795047 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.795013    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" event={"ID":"12fa15c4-7aa4-4664-8a37-10a6f3f2e214","Type":"ContainerStarted","Data":"4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04"}
Oct 30 05:54:17.796918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.796894    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8874-3948c0" event={"ID":"ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5","Type":"ContainerStarted","Data":"5bf9eff7ceb97edb96f9861f348ac53ead3c1540dca081c731bd27b2cbc0c85e"}
Oct 30 05:54:17.896311 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.896266    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"]
Oct 30 05:54:17.901132 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.901105    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:17.914993 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.908002    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podStartSLOduration=2.907984868 podStartE2EDuration="2.907984868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:17.897054485 +0000 UTC m=+1372.728091702" watchObservedRunningTime="2026-10-30 05:54:17.907984868 +0000 UTC m=+1372.739022085"
Oct 30 05:54:17.942579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.942544    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fd1c5d\""
Oct 30 05:54:17.947199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.943906    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2f5f\" (UniqueName: \"kubernetes.io/projected/5c4f5b67-815a-4275-b985-853a93997794-kube-api-access-w2f5f\") pod \"payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh\" (UID: \"5c4f5b67-815a-4275-b985-853a93997794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:17.947199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.944042    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4f5b67-815a-4275-b985-853a93997794-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh\" (UID: \"5c4f5b67-815a-4275-b985-853a93997794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:17.950697 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.950668    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"]
Oct 30 05:54:17.954247 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.954226    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fd1c5d-dockercfg-grjq4\""
Oct 30 05:54:17.996102 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.996013    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:18.164238 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:17.995159    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8874-3948c0" podStartSLOduration=2.995137048 podStartE2EDuration="2.995137048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:17.992619306 +0000 UTC m=+1372.823656523" watchObservedRunningTime="2026-10-30 05:54:17.995137048 +0000 UTC m=+1372.826174266"
Oct 30 05:54:18.265281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.256512    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f5f\" (UniqueName: \"kubernetes.io/projected/5c4f5b67-815a-4275-b985-853a93997794-kube-api-access-w2f5f\") pod \"payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh\" (UID: \"5c4f5b67-815a-4275-b985-853a93997794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:18.265281 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.256684    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4f5b67-815a-4275-b985-853a93997794-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh\" (UID: \"5c4f5b67-815a-4275-b985-853a93997794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:18.311204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.276078    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"]
Oct 30 05:54:18.357200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.344545    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4f5b67-815a-4275-b985-853a93997794-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh\" (UID: \"5c4f5b67-815a-4275-b985-853a93997794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:18.357481 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.352994    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:18.357655 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.357642    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"]
Oct 30 05:54:18.359048 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.353079    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:18.372270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.302150    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:18.372270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.353257    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:18.372270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.365296    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:18.439208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.429384    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2f5f\" (UniqueName: \"kubernetes.io/projected/5c4f5b67-815a-4275-b985-853a93997794-kube-api-access-w2f5f\") pod \"payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh\" (UID: \"5c4f5b67-815a-4275-b985-853a93997794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:18.541169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.541064    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn8c\" (UniqueName: \"kubernetes.io/projected/ef47781f-e949-43ab-9a47-0dadd1a25cb7-kube-api-access-dpn8c\") pod \"payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7\" (UID: \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:18.541578 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.541552    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef47781f-e949-43ab-9a47-0dadd1a25cb7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7\" (UID: \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:18.611036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.580939    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:18.718479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.715031    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn8c\" (UniqueName: \"kubernetes.io/projected/ef47781f-e949-43ab-9a47-0dadd1a25cb7-kube-api-access-dpn8c\") pod \"payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7\" (UID: \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:18.718479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.715129    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef47781f-e949-43ab-9a47-0dadd1a25cb7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7\" (UID: \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:18.718479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.716128    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef47781f-e949-43ab-9a47-0dadd1a25cb7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7\" (UID: \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:18.770776 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:18.770402    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn8c\" (UniqueName: \"kubernetes.io/projected/ef47781f-e949-43ab-9a47-0dadd1a25cb7-kube-api-access-dpn8c\") pod \"payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7\" (UID: \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:19.023792 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.021814    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:19.308440 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.308403    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:19.308636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.308471    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:19.347628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.344252    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:19.449023 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.448988    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"]
Oct 30 05:54:19.482499 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:19.481914    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4f5b67_815a_4275_b985_853a93997794.slice/crio-442bc9704fc988913cb4a2bfac514ba514bc161a8405c0f2a83c2869a1c74fdf WatchSource:0}: Error finding container 442bc9704fc988913cb4a2bfac514ba514bc161a8405c0f2a83c2869a1c74fdf: Status 404 returned error can't find the container with id 442bc9704fc988913cb4a2bfac514ba514bc161a8405c0f2a83c2869a1c74fdf
Oct 30 05:54:19.630765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.630716    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:19.831602 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.831554    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"]
Oct 30 05:54:19.929384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.929350    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh" event={"ID":"5c4f5b67-815a-4275-b985-853a93997794","Type":"ContainerStarted","Data":"442bc9704fc988913cb4a2bfac514ba514bc161a8405c0f2a83c2869a1c74fdf"}
Oct 30 05:54:19.941327 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.939863    5250 generic.go:356] "Generic (PLEG): container finished" podID="a5b67951-a4ed-4e76-822e-07b6b740bbed" containerID="5878625800ae1b0616e774a44d5f3b5dd80f371cef0b7a2d3e81631a7c8e9d41" exitCode=137
Oct 30 05:54:19.942841 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.941552    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68f13eabf544a2135615221ab2c211c63b4a8feb3ead0901c71d15bfca7ac31d"
Oct 30 05:54:19.947645 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:19.947619    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef47781f_e949_43ab_9a47_0dadd1a25cb7.slice/crio-48fe2d581e18952a62ca453183437ebb9e7d648eea8e130a975be873487369c3 WatchSource:0}: Error finding container 48fe2d581e18952a62ca453183437ebb9e7d648eea8e130a975be873487369c3: Status 404 returned error can't find the container with id 48fe2d581e18952a62ca453183437ebb9e7d648eea8e130a975be873487369c3
Oct 30 05:54:19.987234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:19.972409    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:20.063460 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.063426    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8874-3948c0"]
Oct 30 05:54:20.065018 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.064997    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8874-3948c0"]
Oct 30 05:54:20.065837 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.065354    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8874-3948c0" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" containerName="test-curl-8874-3948c0" containerID="cri-o://5bf9eff7ceb97edb96f9861f348ac53ead3c1540dca081c731bd27b2cbc0c85e" gracePeriod=2
Oct 30 05:54:20.066110 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.064552    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b67951-a4ed-4e76-822e-07b6b740bbed-kube-api-access-75ncz\" (UniqueName: \"kubernetes.io/projected/a5b67951-a4ed-4e76-822e-07b6b740bbed-kube-api-access-75ncz\") pod \"a5b67951-a4ed-4e76-822e-07b6b740bbed\" (UID: \"a5b67951-a4ed-4e76-822e-07b6b740bbed\") "
Oct 30 05:54:20.086859 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:20.086475    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-3948c0\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:20.146202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.136723    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b67951-a4ed-4e76-822e-07b6b740bbed-kube-api-access-75ncz" pod "a5b67951-a4ed-4e76-822e-07b6b740bbed" (UID: "a5b67951-a4ed-4e76-822e-07b6b740bbed"). InnerVolumeSpecName "kube-api-access-75ncz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:54:20.217267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.212350    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75ncz\" (UniqueName: \"kubernetes.io/projected/a5b67951-a4ed-4e76-822e-07b6b740bbed-kube-api-access-75ncz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:54:20.225199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.221365    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:20.261386 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:20.241056    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-3948c0\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:20.313208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.306557    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:20.391252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.391212    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:20.596348 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.596152    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:20.972687 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.972645    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh" event={"ID":"5c4f5b67-815a-4275-b985-853a93997794","Type":"ContainerStarted","Data":"12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf"}
Oct 30 05:54:20.974091 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.974067    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:20.976970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.976946    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:20.985862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.982280    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7" event={"ID":"ef47781f-e949-43ab-9a47-0dadd1a25cb7","Type":"ContainerStarted","Data":"bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8"}
Oct 30 05:54:20.985862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.982350    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7" event={"ID":"ef47781f-e949-43ab-9a47-0dadd1a25cb7","Type":"ContainerStarted","Data":"48fe2d581e18952a62ca453183437ebb9e7d648eea8e130a975be873487369c3"}
Oct 30 05:54:20.985862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:20.982380    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:21.102018 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:21.079832    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-3948c0\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:21.124083 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.103103    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh" podStartSLOduration=4.103081581 podStartE2EDuration="4.103081581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:21.042608438 +0000 UTC m=+1375.873645671" watchObservedRunningTime="2026-10-30 05:54:21.103081581 +0000 UTC m=+1375.934118832"
Oct 30 05:54:21.176464 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.175237    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:21.201207 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:21.200432    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-3948c0\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:21.231192 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:21.230924    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-3948c0\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:21.254932 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:21.254444    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8874-b43932\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="a5b67951-a4ed-4e76-822e-07b6b740bbed" pod="opendatahub/test-curl-8874-b43932"
Oct 30 05:54:21.518009 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.517909    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:21.518302 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.518269    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:21.559542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.413164    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:21.592207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.576981    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b67951-a4ed-4e76-822e-07b6b740bbed" path="/var/lib/kubelet/pods/a5b67951-a4ed-4e76-822e-07b6b740bbed/volumes"
Oct 30 05:54:21.609310 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.593672    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:21.645206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.614729    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:54:21.669205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.667589    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:21.753206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.752326    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 05:54:21.803358 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.803257    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:21.894333 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.894288    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:21.936379 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.936325    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:21.937356 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.936266    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:21.946375 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:21.946274    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:22.201748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.190170    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:22.230663 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.230205    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:22.315416 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.315358    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:22.538010 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.489256    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:22.538010 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.489318    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:22.538010 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.509852    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:54:22.560214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.560156    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:22.801259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.708257    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:22.801259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.776383    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 30 05:54:22.801259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.776843    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:22.801259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.777260    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:54:22.982214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.954482    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:22.982214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.973420    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:22.990807 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.990773    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:22.991403 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:22.991384    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:54:23.066261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.058364    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7" podStartSLOduration=6.058344865 podStartE2EDuration="6.058344865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:21.172333546 +0000 UTC m=+1376.003370763" watchObservedRunningTime="2026-10-30 05:54:23.058344865 +0000 UTC m=+1377.889382084"
Oct 30 05:54:23.306839 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.306465    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:23.306839 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.306522    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:23.379370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.377667    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:54:23.419079 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.419051    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:54:23.657210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.635298    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:23.760203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.759353    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:23.972412 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.972310    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:54:23.994258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:23.990197    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:24.020206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.019774    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:24.315259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.314306    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:24.315259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.314449    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:24.410276 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.399919    5250 generic.go:356] "Generic (PLEG): container finished" podID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" containerID="5bf9eff7ceb97edb96f9861f348ac53ead3c1540dca081c731bd27b2cbc0c85e" exitCode=137
Oct 30 05:54:24.439512 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.439461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:24.482201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.479320    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:24.653282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.642275    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"]
Oct 30 05:54:24.653282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.643469    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" containerName="test-curl-8874-3948c0"
Oct 30 05:54:24.653282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.643495    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" containerName="test-curl-8874-3948c0"
Oct 30 05:54:24.653282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.643719    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" containerName="test-curl-8874-3948c0"
Oct 30 05:54:24.653282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.648514    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:24.680731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.680699    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-df544d\""
Oct 30 05:54:24.689245 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.684333    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d646c66-5f08-4dcd-875d-14d997ace3c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p\" (UID: \"1d646c66-5f08-4dcd-875d-14d997ace3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:24.689245 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.684443    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7d6l\" (UniqueName: \"kubernetes.io/projected/1d646c66-5f08-4dcd-875d-14d997ace3c1-kube-api-access-r7d6l\") pod \"payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p\" (UID: \"1d646c66-5f08-4dcd-875d-14d997ace3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:24.712615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.698607    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-df544d-dockercfg-wv5kw\""
Oct 30 05:54:24.712615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.699102    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:54:24.712615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.699451    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 05:54:24.712615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.701939    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:24.712615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.707762    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:54:24.723456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.723427    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:24.796765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.795306    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"]
Oct 30 05:54:24.796765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.796013    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 05:54:24.798361 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.798328    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7d6l\" (UniqueName: \"kubernetes.io/projected/1d646c66-5f08-4dcd-875d-14d997ace3c1-kube-api-access-r7d6l\") pod \"payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p\" (UID: \"1d646c66-5f08-4dcd-875d-14d997ace3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:24.798955 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.798930    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d646c66-5f08-4dcd-875d-14d997ace3c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p\" (UID: \"1d646c66-5f08-4dcd-875d-14d997ace3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:24.811204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.800031    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d646c66-5f08-4dcd-875d-14d997ace3c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p\" (UID: \"1d646c66-5f08-4dcd-875d-14d997ace3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:24.999262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.972969    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5-kube-api-access-rhrrh\" (UniqueName: \"kubernetes.io/projected/ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5-kube-api-access-rhrrh\") pod \"ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5\" (UID: \"ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5\") "
Oct 30 05:54:25.008690 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:24.798398    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 05:54:25.430205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.055501    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:25.457646 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.065753    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 05:54:25.501730 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.170223    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:25.822225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.381650    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:25.841387 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.841344    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:25.841606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.699494    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:25.841738 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.746327    5250 scope.go:122] "RemoveContainer" containerID="269dfb650e5f95643fa7f2da5e734573dd6fcf7d95d36babe381f1a6a3def24d"
Oct 30 05:54:25.841992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.528119    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:25.869314 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.819456    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:25.869600 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.819548    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:25.869822 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.385568    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5-kube-api-access-rhrrh" pod "ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" (UID: "ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5"). InnerVolumeSpecName "kube-api-access-rhrrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:54:25.963206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.763612    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:25.979791 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:25.960761    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7d6l\" (UniqueName: \"kubernetes.io/projected/1d646c66-5f08-4dcd-875d-14d997ace3c1-kube-api-access-r7d6l\") pod \"payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p\" (UID: \"1d646c66-5f08-4dcd-875d-14d997ace3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:26.060204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.045102    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-df544d-dockercfg-wv5kw\""
Oct 30 05:54:26.073268 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.063325    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8874-3948c0"
Oct 30 05:54:26.118203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.076463    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:26.288698 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.102535    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrrh\" (UniqueName: \"kubernetes.io/projected/ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5-kube-api-access-rhrrh\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:54:26.288912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.110132    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5" path="/var/lib/kubelet/pods/ccc41d28-79c9-4c1d-96cc-426e3a4b4cd5/volumes"
Oct 30 05:54:26.290690 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.290665    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:54:26.290916 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.290901    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:54:26.291051 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.291035    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"]
Oct 30 05:54:26.295089 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.295070    5250 scope.go:122] "RemoveContainer" containerID="5bf9eff7ceb97edb96f9861f348ac53ead3c1540dca081c731bd27b2cbc0c85e"
Oct 30 05:54:26.295341 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.177383    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:26.300230 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.180340    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:26.341373 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.341261    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 05:54:26.341641 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.341610    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 05:54:26.364746 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.364708    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"]
Oct 30 05:54:26.374463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.365107    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:54:26.374753 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.374733    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:54:26.402209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.375866    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:26.402475 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.399467    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:26.481050 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.431349    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:26.513217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.502720    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:26.601508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.509907    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1390fad0-655d-4660-a34f-4a2977f468ef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df544d-77949d54b4-stbxx\" (UID: \"1390fad0-655d-4660-a34f-4a2977f468ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:26.602638 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.602609    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52cg\" (UniqueName: \"kubernetes.io/projected/1390fad0-655d-4660-a34f-4a2977f468ef-kube-api-access-g52cg\") pod \"payload-processing-e2e-api-a-df544d-77949d54b4-stbxx\" (UID: \"1390fad0-655d-4660-a34f-4a2977f468ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:26.603341 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.568932    5250 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:26.619718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.573293    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:26.675421 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.669346    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:54:26.704748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.704706    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52cg\" (UniqueName: \"kubernetes.io/projected/1390fad0-655d-4660-a34f-4a2977f468ef-kube-api-access-g52cg\") pod \"payload-processing-e2e-api-a-df544d-77949d54b4-stbxx\" (UID: \"1390fad0-655d-4660-a34f-4a2977f468ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:26.705144 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.705116    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1390fad0-655d-4660-a34f-4a2977f468ef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df544d-77949d54b4-stbxx\" (UID: \"1390fad0-655d-4660-a34f-4a2977f468ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:26.706653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.706628    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1390fad0-655d-4660-a34f-4a2977f468ef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df544d-77949d54b4-stbxx\" (UID: \"1390fad0-655d-4660-a34f-4a2977f468ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:26.708963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.708942    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 05:54:26.799207 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:26.757054    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 05:54:26.923215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:26.923166    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 05:54:27.018632 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.018605    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 05:54:27.042583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.042354    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:54:27.042839 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.042820    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 05:54:27.130348 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.120331    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52cg\" (UniqueName: \"kubernetes.io/projected/1390fad0-655d-4660-a34f-4a2977f468ef-kube-api-access-g52cg\") pod \"payload-processing-e2e-api-a-df544d-77949d54b4-stbxx\" (UID: \"1390fad0-655d-4660-a34f-4a2977f468ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:27.140393 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.140257    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 05:54:27.195061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.194786    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:27.317395 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.317363    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:27.318260 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.318240    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:27.319148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.319130    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:54:27.500673 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.500400    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:27.510729 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.510334    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"]
Oct 30 05:54:27.616417 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.615976    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"]
Oct 30 05:54:27.686921 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:27.683656    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:28.031897 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.031865    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx" event={"ID":"1390fad0-655d-4660-a34f-4a2977f468ef","Type":"ContainerStarted","Data":"895b13eb75eda9a4e117f1fbeb85813c7f0083d049c831974086e14dbb482f91"}
Oct 30 05:54:28.032094 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.031916    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx" event={"ID":"1390fad0-655d-4660-a34f-4a2977f468ef","Type":"ContainerStarted","Data":"c5cb683a3dd64eb5e3cb2be25d8e513c0d08528aa06be8979a6161f87bbb9003"}
Oct 30 05:54:28.032094 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.031955    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:28.033695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.033671    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p" event={"ID":"1d646c66-5f08-4dcd-875d-14d997ace3c1","Type":"ContainerStarted","Data":"44f6817ba80abd913d5f8ea2809231ecc460b2464ab5a332c004d3f7d202c447"}
Oct 30 05:54:28.033831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.033713    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p" event={"ID":"1d646c66-5f08-4dcd-875d-14d997ace3c1","Type":"ContainerStarted","Data":"f5ed72fc78fc0b47efd1f91aceb2b5a495dfa3b0999128d6e2cbb87be42062af"}
Oct 30 05:54:28.033886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.033838    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:28.084769 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.084706    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx" podStartSLOduration=4.084685449 podStartE2EDuration="4.084685449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:28.080679352 +0000 UTC m=+1382.911716570" watchObservedRunningTime="2026-10-30 05:54:28.084685449 +0000 UTC m=+1382.915722667"
Oct 30 05:54:28.090288 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:28.090254    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache]"
Oct 30 05:54:28.102970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.102946    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:28.123947 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:28.123901    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p" podStartSLOduration=4.123888317 podStartE2EDuration="4.123888317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:28.122613713 +0000 UTC m=+1382.953650932" watchObservedRunningTime="2026-10-30 05:54:28.123888317 +0000 UTC m=+1382.954925533"
Oct 30 05:54:29.346699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.345823    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:29.543249 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.543093    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"]
Oct 30 05:54:29.549937 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.549817    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.591499 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.591437    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"]
Oct 30 05:54:29.619391 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.619293    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-37bbcc-openshift-default-dockercfg-h7hzk\""
Oct 30 05:54:29.633136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.633072    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:29.693707 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.693670    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.693900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.693738    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.693900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.693804    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.694033 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.693934    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.694033 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.693997    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.694143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.694040    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.694143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.694078    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.694143 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.694117    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68xm\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-kube-api-access-c68xm\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.694371 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.694170    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796114    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796165    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796221    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796251    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68xm\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-kube-api-access-c68xm\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796287    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796462    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796573    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796642    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.796883    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.798938    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.801213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.800026    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.803566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.801810    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.803566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.802078    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.803885 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.803833    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.804140 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.804097    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.807747 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.807702    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.828858 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.828782    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68xm\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-kube-api-access-c68xm\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.830644 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.830581    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:29.880842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:29.880749    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:30.259208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:30.259161    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"]
Oct 30 05:54:30.260647 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:30.260617    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd77eb20_958f_4ff7_b4c5_d0ed079b3ff6.slice/crio-1563dfa24bd01ace8449932993c90cf4c177dc73d74b217f7788c5d050ef2b63 WatchSource:0}: Error finding container 1563dfa24bd01ace8449932993c90cf4c177dc73d74b217f7788c5d050ef2b63: Status 404 returned error can't find the container with id 1563dfa24bd01ace8449932993c90cf4c177dc73d74b217f7788c5d050ef2b63
Oct 30 05:54:30.264441 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:30.264401    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:30.264524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:30.264499    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:30.264563 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:30.264545    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:30.526532 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:30.526445    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:31.095206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:31.091957    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" event={"ID":"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6","Type":"ContainerStarted","Data":"f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea"}
Oct 30 05:54:31.095206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:31.092021    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" event={"ID":"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6","Type":"ContainerStarted","Data":"1563dfa24bd01ace8449932993c90cf4c177dc73d74b217f7788c5d050ef2b63"}
Oct 30 05:54:31.179200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:31.178210    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" podStartSLOduration=2.178163188 podStartE2EDuration="2.178163188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:31.152660805 +0000 UTC m=+1385.983698024" watchObservedRunningTime="2026-10-30 05:54:31.178163188 +0000 UTC m=+1386.009200405"
Oct 30 05:54:31.881470 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:31.881429    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:31.883612 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:31.883582    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:54:31.883737 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:31.883632    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:54:32.006023 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:32.005989    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:54:32.281209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:32.279851    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:32.412246 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:32.412150    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:54:32.702003 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:32.701963    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:32.881863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:32.881611    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:54:32.881863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:32.881659    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:54:33.115752 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.115722    5250 generic.go:356] "Generic (PLEG): container finished" podID="389eaf27-0be7-44d2-8e5e-01986e050701" containerID="649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726" exitCode=0
Oct 30 05:54:33.115932 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.115795    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" event={"ID":"389eaf27-0be7-44d2-8e5e-01986e050701","Type":"ContainerDied","Data":"649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726"}
Oct 30 05:54:33.116407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.116386    5250 scope.go:122] "RemoveContainer" containerID="649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726"
Oct 30 05:54:33.118076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.118042    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:33.118207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.118133    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:33.118207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.118201    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:54:33.124590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.124573    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:54:33.776962 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.768939    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:33.892202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.891539    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:54:33.892202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:33.891597    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:54:34.125505 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:34.125374    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" event={"ID":"389eaf27-0be7-44d2-8e5e-01986e050701","Type":"ContainerStarted","Data":"c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b"}
Oct 30 05:54:34.610396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:34.610358    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:54:34.611782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:34.611754    5250 patch_prober.go:29] interesting pod/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 05:54:34.611921 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:34.611802    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 05:54:34.776205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:34.775706    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:54:34.884795 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:34.881577    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:54:34.884795 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:34.881642    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:54:35.186261 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:35.180730    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:35.610536 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:35.610451    5250 patch_prober.go:29] interesting pod/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 05:54:35.610536 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:35.610514    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 05:54:35.887202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:35.885576    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:54:35.887202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:35.885652    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:54:36.610783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:36.610744    5250 patch_prober.go:29] interesting pod/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 05:54:36.611403 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:36.611372    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 05:54:36.916888 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:36.916846    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache]"
Oct 30 05:54:36.935220 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:36.934497    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:36.937242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:36.935627    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:36.937242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:36.936634    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 05:54:36.990730 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:36.990688    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:37.610790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:37.610760    5250 patch_prober.go:29] interesting pod/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 05:54:37.611199 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:37.610816    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 05:54:38.610401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:38.610365    5250 patch_prober.go:29] interesting pod/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 05:54:38.610581 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:38.610419    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 05:54:38.647359 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:38.647325    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:39.045612 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:39.045584    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:54:39.046000 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:39.045977    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:54:39.615239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:39.615171    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:54:39.616432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:39.616407    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:54:40.459689 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:40.459654    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:42.313922 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.313880    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"]
Oct 30 05:54:42.319922 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.319892    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.337914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.337888    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fd1c5d-serving-cert\""
Oct 30 05:54:42.349624 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.349596    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"]
Oct 30 05:54:42.356195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.351021    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rs86\" (UniqueName: \"kubernetes.io/projected/5d769840-311e-429d-9aba-4533ee9f66b7-kube-api-access-2rs86\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.356195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.351080    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-maas-api-tls\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.356195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.351197    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-metrics-certs\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.356195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.351533    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d769840-311e-429d-9aba-4533ee9f66b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.421822 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.421784    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:42.452780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.452748    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d769840-311e-429d-9aba-4533ee9f66b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.452964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.452791    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rs86\" (UniqueName: \"kubernetes.io/projected/5d769840-311e-429d-9aba-4533ee9f66b7-kube-api-access-2rs86\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.452964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.452813    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-maas-api-tls\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.452964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.452874    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-metrics-certs\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.453537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.453511    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d769840-311e-429d-9aba-4533ee9f66b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.455490 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.455471    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-maas-api-tls\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.455555 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.455492    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-metrics-certs\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.477424 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.477398    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rs86\" (UniqueName: \"kubernetes.io/projected/5d769840-311e-429d-9aba-4533ee9f66b7-kube-api-access-2rs86\") pod \"maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.569513 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.569417    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:42.641456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.641426    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:42.974744 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:42.974709    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d769840_311e_429d_9aba_4533ee9f66b7.slice/crio-7f0d73ab11e3b259b9c2ae3545aaf324401b2ffadc14c7385ed3dee91c9f1010 WatchSource:0}: Error finding container 7f0d73ab11e3b259b9c2ae3545aaf324401b2ffadc14c7385ed3dee91c9f1010: Status 404 returned error can't find the container with id 7f0d73ab11e3b259b9c2ae3545aaf324401b2ffadc14c7385ed3dee91c9f1010
Oct 30 05:54:42.975076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:42.975053    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"]
Oct 30 05:54:43.146270 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:43.146229    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache]"
Oct 30 05:54:43.229262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:43.229090    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz" event={"ID":"5d769840-311e-429d-9aba-4533ee9f66b7","Type":"ContainerStarted","Data":"7f0d73ab11e3b259b9c2ae3545aaf324401b2ffadc14c7385ed3dee91c9f1010"}
Oct 30 05:54:44.238436 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:44.238383    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz" event={"ID":"5d769840-311e-429d-9aba-4533ee9f66b7","Type":"ContainerStarted","Data":"547e9ebf131cda429e8beea6ff89a906134b29cfc0d86b2646e69e6a887b55f8"}
Oct 30 05:54:44.238885 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:44.238571    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:44.286026 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:44.285975    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz" podStartSLOduration=1.910814526 podStartE2EDuration="2.285959382s" totalImagesPullingTime="375.144856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:44.282676543 +0000 UTC m=+1399.113713799" watchObservedRunningTime="2026-10-30 05:54:44.285959382 +0000 UTC m=+1399.116996598"
Oct 30 05:54:44.313756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:44.313649    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:45.929986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:45.929950    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:46.936229 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:46.936156    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache]"
Oct 30 05:54:47.442482 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:47.442434    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:47.618750 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:47.618716    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:47.913566 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:47.913327    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:49.033522 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.033490    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"]
Oct 30 05:54:49.046989 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.046961    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09b5967-5scmp"]
Oct 30 05:54:49.287718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.287623    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8703bc1f-eb3b-4e3d-9a02-f406819a25ea" path="/var/lib/kubelet/pods/8703bc1f-eb3b-4e3d-9a02-f406819a25ea/volumes"
Oct 30 05:54:49.916049 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.916019    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"]
Oct 30 05:54:49.921018 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.920996    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:49.941551 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.941527    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-df544d-serving-cert\""
Oct 30 05:54:49.945633 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.945609    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"]
Oct 30 05:54:49.996275 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.996244    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:49.996460 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.996442    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c44db-9f95-42b5-bac6-9a827a1fd006-openshift-service-ca\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:49.996583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.996570    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-metrics-certs\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:49.996735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:49.996721    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5974\" (UniqueName: \"kubernetes.io/projected/9c1c44db-9f95-42b5-bac6-9a827a1fd006-kube-api-access-t5974\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.037065 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.037024    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:50.099510 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.099465    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5974\" (UniqueName: \"kubernetes.io/projected/9c1c44db-9f95-42b5-bac6-9a827a1fd006-kube-api-access-t5974\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.099750 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.099638    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.099836 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:50.099782    5250 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-serving-cert: secret "maas-api-e2e-api-a-df544d-serving-cert" not found
Oct 30 05:54:50.099909 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:50.099860    5250 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls podName:9c1c44db-9f95-42b5-bac6-9a827a1fd006 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:50.599838599 +0000 UTC m=+1405.430875808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls") pod "maas-api-e2e-api-a-df544d-66fd667b99-l6hd6" (UID: "9c1c44db-9f95-42b5-bac6-9a827a1fd006") : secret "maas-api-e2e-api-a-df544d-serving-cert" not found
Oct 30 05:54:50.100283 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.100253    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c44db-9f95-42b5-bac6-9a827a1fd006-openshift-service-ca\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.100437 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.100411    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-metrics-certs\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.101724 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.101701    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c44db-9f95-42b5-bac6-9a827a1fd006-openshift-service-ca\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.103535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.103514    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-metrics-certs\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.126146 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.126111    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5974\" (UniqueName: \"kubernetes.io/projected/9c1c44db-9f95-42b5-bac6-9a827a1fd006-kube-api-access-t5974\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.179924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.179823    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:50.252656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.252628    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:54:50.438741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.438636    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:50.610283 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.610243    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.612875 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.612850    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls\") pod \"maas-api-e2e-api-a-df544d-66fd667b99-l6hd6\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:50.840629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:50.840540    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:51.200456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:51.200429    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"]
Oct 30 05:54:51.201010 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:54:51.200987    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1c44db_9f95_42b5_bac6_9a827a1fd006.slice/crio-a621f83534a191c0f5cddf04674d913703b5d5214269022517986f73848e47e4 WatchSource:0}: Error finding container a621f83534a191c0f5cddf04674d913703b5d5214269022517986f73848e47e4: Status 404 returned error can't find the container with id a621f83534a191c0f5cddf04674d913703b5d5214269022517986f73848e47e4
Oct 30 05:54:51.306411 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:51.306375    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6" event={"ID":"9c1c44db-9f95-42b5-bac6-9a827a1fd006","Type":"ContainerStarted","Data":"a621f83534a191c0f5cddf04674d913703b5d5214269022517986f73848e47e4"}
Oct 30 05:54:52.230941 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:52.230908    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:52.316278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:52.316244    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6" event={"ID":"9c1c44db-9f95-42b5-bac6-9a827a1fd006","Type":"ContainerStarted","Data":"21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1"}
Oct 30 05:54:52.316461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:52.316384    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:52.370083 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:52.370029    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6" podStartSLOduration=2.900540678 podStartE2EDuration="3.370012975s" totalImagesPullingTime="469.472297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:52.366899451 +0000 UTC m=+1407.197936671" watchObservedRunningTime="2026-10-30 05:54:52.370012975 +0000 UTC m=+1407.201050192"
Oct 30 05:54:53.583701 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:53.583670    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:54.256377 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:54.256341    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:54.538940 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:54.538853    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:54.937489 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:54.937456    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:55.081912 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:55.081876    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:55.238711 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:55.238568    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:55.483138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:55.483105    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:57.022441 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:57.022372    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 05:54:57.068132 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:57.068094    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:58.089456 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:54:58.089417    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache]"
Oct 30 05:54:58.338866 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:58.338839    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:54:58.823674 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:54:58.823638    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:00.264465 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:00.264429    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:01.448950 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:55:01.448901    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache]"
Oct 30 05:55:01.449466 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:55:01.448964    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache]"
Oct 30 05:55:01.788557 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:01.788472    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:02.358102 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:02.358059    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:02.386314 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:55:02.386257    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:55:02.501468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:02.501437    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:02.703334 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:02.703301    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:03.679931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:03.679827    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:04.010167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:04.010079    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:04.639675 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:04.639639    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:05.529598 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:05.529564    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:05.681842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:05.681804    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:07.110350 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:55:07.110311    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 05:55:07.221103 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:07.221064    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:08.471605 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:08.471576    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:08.695835 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:08.695806    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:09.656572 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:09.656538    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:10.240208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:10.240155    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:10.453159 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:10.453128    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:11.220537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:11.220507    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:11.580869 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:11.580773    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:12.031200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:12.031158    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"]
Oct 30 05:55:12.047513 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:12.047483    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a78db-bprfv"]
Oct 30 05:55:13.015815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:13.015772    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:13.096534 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:55:13.096490    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 05:55:13.284093 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:13.284013    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c82d544-c33b-4567-a811-4e21a7bf975a" path="/var/lib/kubelet/pods/3c82d544-c33b-4567-a811-4e21a7bf975a/volumes"
Oct 30 05:55:15.440322 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:15.440172    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:15.864834 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:15.864754    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:16.754522 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:16.754489    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:17.252876 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:55:17.252835    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc41d28_79c9_4c1d_96cc_426e3a4b4cd5.slice/crio-10015659062b3cffaaa076b5d05d56a56696dc03931b9a93cebbc3d31b954c72\": RecentStats: unable to find data in memory cache]"
Oct 30 05:55:17.609453 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:17.609344    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:18.057117 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:18.057080    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:18.949965 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:18.949928    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:19.380976 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:19.380863    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:21.791399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:21.791366    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:23.298833 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:23.298803    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:23.469933 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:23.469905    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:24.632571 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:24.632534    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:25.313234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:25.313204    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:25.525284 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:25.525253    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:27.199518 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:27.199489    5250 scope.go:122] "RemoveContainer" containerID="35957b6bb5c1877c5294a2ac0b5a313d7fa2671d4e6120731955e6565dbccad7"
Oct 30 05:55:27.210803 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:27.210773    5250 scope.go:122] "RemoveContainer" containerID="62b2505d60b75527eff17d5fcd4522357edb391a9a0dd30286b52b3a262e2584"
Oct 30 05:55:27.524013 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:27.523916    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:28.477234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:28.477189    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:29.141870 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:29.141829    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:29.511485 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:29.511389    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:29.717206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:29.710882    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:29.865226 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:29.865100    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:30.437342 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:30.437301    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:31.233635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:31.233594    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:31.422369 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:31.422338    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:32.066201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.061347    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"]
Oct 30 05:55:32.073734 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.073705    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.096651 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.096621    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-df544d-openshift-default-dockercfg-6dwdl\""
Oct 30 05:55:32.099500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.099468    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35668233-ef08-4087-a61d-f788c405e7d9-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.099715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.099696    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5hg\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-kube-api-access-cb5hg\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.099837 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.099822    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.099946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.099932    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.100074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.100060    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.100318 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.100291    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.100468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.100454    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.100695 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.100678    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.100802 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.100788    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35668233-ef08-4087-a61d-f788c405e7d9-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.105200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.102346    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"]
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213395    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213477    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213654    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213686    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35668233-ef08-4087-a61d-f788c405e7d9-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213717    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35668233-ef08-4087-a61d-f788c405e7d9-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213754    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5hg\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-kube-api-access-cb5hg\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213800    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213843    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.213908    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.217327    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35668233-ef08-4087-a61d-f788c405e7d9-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.247784    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.273202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.247990    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.324990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.324778    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.325429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.325400    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.328244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.328218    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35668233-ef08-4087-a61d-f788c405e7d9-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.328755 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.328465    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5hg\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-kube-api-access-cb5hg\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.328755 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.328712    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.329305 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.329279    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-h8sfd\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.380919 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:55:32.380422    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:55:32.517535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.517503    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:32.892352 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.892327    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"]
Oct 30 05:55:32.892578 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:55:32.892348    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35668233_ef08_4087_a61d_f788c405e7d9.slice/crio-c719da2606465e620d1b9ac4f81899d099d7e15aa48f2f0e21749dbdaa212790 WatchSource:0}: Error finding container c719da2606465e620d1b9ac4f81899d099d7e15aa48f2f0e21749dbdaa212790: Status 404 returned error can't find the container with id c719da2606465e620d1b9ac4f81899d099d7e15aa48f2f0e21749dbdaa212790
Oct 30 05:55:32.895335 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.895296    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:32.895454 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.895379    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:32.895454 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.895423    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:32.967433 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:32.967395    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:33.115864 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:33.113768    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:33.751736 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:33.751698    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:33.818914 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:33.818876    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"]
Oct 30 05:55:33.819154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:33.819123    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy" containerID="cri-o://c2abb731be8c04f5245eb52a7d09b9e70ce68b20c21834c04045ef999da8a609" gracePeriod=30
Oct 30 05:55:33.884429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:33.883997    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" event={"ID":"35668233-ef08-4087-a61d-f788c405e7d9","Type":"ContainerStarted","Data":"4563e0429fd9ae42832851a6df25065881e91ac444c603d870300fc949f9833f"}
Oct 30 05:55:33.884429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:33.884110    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" event={"ID":"35668233-ef08-4087-a61d-f788c405e7d9","Type":"ContainerStarted","Data":"c719da2606465e620d1b9ac4f81899d099d7e15aa48f2f0e21749dbdaa212790"}
Oct 30 05:55:33.943463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:33.943403    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" podStartSLOduration=1.943385711 podStartE2EDuration="1.943385711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:33.937058559 +0000 UTC m=+1448.768095776" watchObservedRunningTime="2026-10-30 05:55:33.943385711 +0000 UTC m=+1448.774422933"
Oct 30 05:55:34.517996 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:34.517953    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:34.520005 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:34.519976    5250 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 05:55:34.520155 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:34.520022    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 05:55:34.529127 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:34.529080    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:35.168883 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.168797    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:35.231213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.231151    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"]
Oct 30 05:55:35.279294 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.263765    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"]
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.331515    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-credential-socket\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.361600    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istiod-ca-cert\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.361696    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gb9\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-kube-api-access-69gb9\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.361746    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-token\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.361793    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-socket\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.361903    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-podinfo\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.361944    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-envoy\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.362084    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-certs\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.372782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.362289    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-data\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.450653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.450552    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.477205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.454351    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:35.477516 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.474820    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gb9\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-kube-api-access-69gb9\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.477703 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.477686    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-token\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.477846 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.477829    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-socket\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.478057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.478037    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-podinfo\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.478198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.478164    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-envoy\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.478499 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.478478    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-certs\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.478691 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.478662    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-socket\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.500341 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.500306    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-data\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.500619 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.500599    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-credential-socket\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.500785 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.500768    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istiod-ca-cert\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.514519 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.513607    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-certs\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.514519 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.514246    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istiod-ca-cert\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.514970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.514949    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-credential-socket\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.533318 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.529888    5250 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 05:55:35.533614 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.533589    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 05:55:35.533724 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.531785    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-data\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.537375 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.537350    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-76b905-openshift-default-dockercfg-f62v5\""
Oct 30 05:55:35.538783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.538730    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-envoy\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.541576 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.541549    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-podinfo\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.572198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.566551    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gb9\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-kube-api-access-69gb9\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.596472 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.596222    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-token\") pod \"e2e-api-b-76b905-openshift-default-686f48485c-p7ttx\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") " pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:35.643354 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.643321    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:35.848370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.834992    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:35.848370 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:35.847758    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:36.101683 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.101560    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:36.285092 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.285059    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:36.395553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.395519    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"]
Oct 30 05:55:36.401273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.401252    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.414989 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.414950    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"]
Oct 30 05:55:36.421016 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.420230    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.435499 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.435476    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e94bd8-openshift-default-dockercfg-nttsj\""
Oct 30 05:55:36.444198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.443248    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"]
Oct 30 05:55:36.446608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.446587    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"]
Oct 30 05:55:36.494568 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.494527    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65ds\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-kube-api-access-m65ds\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.494772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.494624    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.494772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.494677    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfxs\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-kube-api-access-qlfxs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.494884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.494824    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.494956 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.494933    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.495250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495151    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.495344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495300    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.495344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495338    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.495454 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495363    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.495454 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495408    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.495564 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495529    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.495618 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495584    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.495675 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495635    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.495735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495692    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.495735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495722    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8de185f-7e92-432a-b994-6673f21895de-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.495858 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495831    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8de185f-7e92-432a-b994-6673f21895de-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.495963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495927    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.496036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.495971    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.518147 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.518124    5250 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 05:55:36.518300 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.518162    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 05:55:36.532900 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:55:36.532875    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d3b064_1ad9_48ab_b1f4_9b975b1a321a.slice/crio-4699f561e81946814f728b0a061a4ce9439e5405b414d9b24b5dbe9c4fab5f0a WatchSource:0}: Error finding container 4699f561e81946814f728b0a061a4ce9439e5405b414d9b24b5dbe9c4fab5f0a: Status 404 returned error can't find the container with id 4699f561e81946814f728b0a061a4ce9439e5405b414d9b24b5dbe9c4fab5f0a
Oct 30 05:55:36.533856 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.533830    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"]
Oct 30 05:55:36.536047 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.536017    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:36.536116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.536082    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:36.536116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.536112    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:36.597465 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.597431    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.598078 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598060    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.598326 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598293    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.598326 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598327    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.598508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598356    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.598508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598384    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.598508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598412    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.598508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598432    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.598508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598463    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.598508 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598498    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.598836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598564    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.598836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598592    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8de185f-7e92-432a-b994-6673f21895de-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.598836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598672    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8de185f-7e92-432a-b994-6673f21895de-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.598836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598741    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.598836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598803    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.599112 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598865    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65ds\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-kube-api-access-m65ds\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.599112 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.598950    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.599112 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.599005    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfxs\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-kube-api-access-qlfxs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.599112 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.599060    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.599383 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.599140    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.599383 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.597978    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.601233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.599875    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.601233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.599970    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.601233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.600685    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8de185f-7e92-432a-b994-6673f21895de-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.601233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.600892    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.601233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.601060    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.601233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.601163    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.601622 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.601307    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.604987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.604798    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.606670 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.606644    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.606779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.606744    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8de185f-7e92-432a-b994-6673f21895de-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.609957 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.608844    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.630606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.630420    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.630606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.630545    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfxs\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-kube-api-access-qlfxs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:36.630606 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.630562    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65ds\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-kube-api-access-m65ds\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.632076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.632056    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.719747 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.719716    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:36.748012 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:36.747980    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:37.112874 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.112118    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"]
Oct 30 05:55:37.113695 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:55:37.113663    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8de185f_7e92_432a_b994_6673f21895de.slice/crio-295108b0333be1caa0ef9a3f1758396de77d96b072c2adec905b06951b1a3aed WatchSource:0}: Error finding container 295108b0333be1caa0ef9a3f1758396de77d96b072c2adec905b06951b1a3aed: Status 404 returned error can't find the container with id 295108b0333be1caa0ef9a3f1758396de77d96b072c2adec905b06951b1a3aed
Oct 30 05:55:37.116430 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.116398    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:37.116531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.116462    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:37.116531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.116491    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:37.122986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.122490    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" event={"ID":"46d3b064-1ad9-48ab-b1f4-9b975b1a321a","Type":"ContainerStarted","Data":"2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989"}
Oct 30 05:55:37.122986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.122545    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" event={"ID":"46d3b064-1ad9-48ab-b1f4-9b975b1a321a","Type":"ContainerStarted","Data":"4699f561e81946814f728b0a061a4ce9439e5405b414d9b24b5dbe9c4fab5f0a"}
Oct 30 05:55:37.220982 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.220936    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"]
Oct 30 05:55:37.228885 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.228827    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podStartSLOduration=3.228808712 podStartE2EDuration="3.228808712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:37.196893988 +0000 UTC m=+1452.027931207" watchObservedRunningTime="2026-10-30 05:55:37.228808712 +0000 UTC m=+1452.059845926"
Oct 30 05:55:37.235891 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.235851    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:37.236007 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.235934    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:37.236007 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.235981    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:55:37.288931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.288859    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:37.441888 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.441820    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:37.521288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.518713    5250 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 05:55:37.521288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.518770    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 05:55:37.855461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.855286    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:37.911526 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:37.911491    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:38.151201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.146232    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" event={"ID":"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba","Type":"ContainerStarted","Data":"ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76"}
Oct 30 05:55:38.151201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.146298    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" event={"ID":"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba","Type":"ContainerStarted","Data":"3d14d48eb4aff2778e5e6aac8eeec9e618c8d6d4c719a0865692633a946b8d98"}
Oct 30 05:55:38.151479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.151245    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" event={"ID":"a8de185f-7e92-432a-b994-6673f21895de","Type":"ContainerStarted","Data":"d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e"}
Oct 30 05:55:38.151479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.151460    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" event={"ID":"a8de185f-7e92-432a-b994-6673f21895de","Type":"ContainerStarted","Data":"295108b0333be1caa0ef9a3f1758396de77d96b072c2adec905b06951b1a3aed"}
Oct 30 05:55:38.151590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.151507    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:38.154207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.153543    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:55:38.221132 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.221077    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podStartSLOduration=2.22105519 podStartE2EDuration="2.22105519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:38.207421099 +0000 UTC m=+1453.038458315" watchObservedRunningTime="2026-10-30 05:55:38.22105519 +0000 UTC m=+1453.052092412"
Oct 30 05:55:38.530317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.519534    5250 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 05:55:38.530317 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.519923    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 05:55:38.729270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.729173    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:38.731348 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.731312    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 05:55:38.731489 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.731379    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 05:55:38.754378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.754346    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:38.754703 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.754669    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 05:55:38.754809 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.754727    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 05:55:38.793840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:38.793739    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:39.170220 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.169852    5250 generic.go:356] "Generic (PLEG): container finished" podID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerID="c2abb731be8c04f5245eb52a7d09b9e70ce68b20c21834c04045ef999da8a609" exitCode=0
Oct 30 05:55:39.175282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.171169    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" event={"ID":"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc","Type":"ContainerDied","Data":"c2abb731be8c04f5245eb52a7d09b9e70ce68b20c21834c04045ef999da8a609"}
Oct 30 05:55:39.563648 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.543496    5250 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 05:55:39.565346 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.565309    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 05:55:39.778782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.778691    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 05:55:39.779088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.779055    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 05:55:39.812213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.794448    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:55:39.832210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.822709    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podStartSLOduration=3.822686395 podStartE2EDuration="3.822686395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:38.416577698 +0000 UTC m=+1453.247614917" watchObservedRunningTime="2026-10-30 05:55:39.822686395 +0000 UTC m=+1454.653723612"
Oct 30 05:55:39.870278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.854217    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 05:55:39.870278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.854309    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 05:55:39.870640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.870611    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"]
Oct 30 05:55:39.871906 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.871875    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy"
Oct 30 05:55:39.872086 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.872067    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy"
Oct 30 05:55:39.872492 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:39.872473    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" containerName="istio-proxy"
Oct 30 05:55:40.181205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.175443    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"]
Oct 30 05:55:40.181205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.176473    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:40.201214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.198902    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.210658    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.210855    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.210975    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-credential-socket\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.211024    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.211056    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-data\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.211087    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istiod-ca-cert\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.211131    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-socket\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.211277    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-kube-api-access-znd85\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-kube-api-access-znd85\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.218201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.211394    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-certs\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:40.267242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.245758    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwzt\" (UniqueName: \"kubernetes.io/projected/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-kube-api-access-kwwzt\") pod \"payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk\" (UID: \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:40.267817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.267779    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk\" (UID: \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:40.415473 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.405573    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4323f6-dockercfg-dh9sg\""
Oct 30 05:55:40.439771 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.439681    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4323f6\""
Oct 30 05:55:40.505272 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.505238    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-data" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:55:40.509537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.509505    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istiod-ca-cert" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:55:40.546771 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.543449    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-socket" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:55:40.546944 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.546733    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:40.700027 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.616478    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-certs" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:55:40.701066 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.618987    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-credential-socket" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:55:40.701169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.679471    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-kube-api-access-znd85" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "kube-api-access-znd85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:55:40.701270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.666697    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:55:40.701979 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.666851    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:55:40.796706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.796660    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:55:40.819488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.819449    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 05:55:40.819673 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.819506    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 05:55:40.823837 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.822473    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:40.824169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.824145    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 05:55:40.824374 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.824349    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 05:55:40.857754 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.857673    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"
Oct 30 05:55:40.905366 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.897725    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq" event={"ID":"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc","Type":"ContainerDied","Data":"f21913a47db097a1b9b8e6b5b10a7c8c2b75e2c6497c661d99abaa89a0262620"}
Oct 30 05:55:40.923352 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.923261    5250 scope.go:122] "RemoveContainer" containerID="c2abb731be8c04f5245eb52a7d09b9e70ce68b20c21834c04045ef999da8a609"
Oct 30 05:55:40.937860 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.932598    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:55:40.983300 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:40.970874    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:55:41.015248 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.001135    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:55:41.024674 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.024630    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:55:41.044984 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.044947    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 05:55:41.067205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.062278    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:55:41.067416 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.067203    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:41.067416 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.067369    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:41.067547 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.067503    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy\") pod \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\" (UID: \"4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc\") "
Oct 30 05:55:41.092313 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.068031    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwzt\" (UniqueName: \"kubernetes.io/projected/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-kube-api-access-kwwzt\") pod \"payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk\" (UID: \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:41.092702 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.092666    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk\" (UID: \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:41.110595 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:55:41.068663    5250 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc/volumes/kubernetes.io~empty-dir/istio-envoy
Oct 30 05:55:41.139391 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.139357    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:55:41.139591 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.124565    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:41.141286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.085377    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:55:41.143223 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:55:41.068726    5250 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 30 05:55:41.144642 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.144620    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:55:41.144771 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.137545    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.126218    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk\" (UID: \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.124984    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znd85\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-kube-api-access-znd85\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.146165    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.146209    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.146224    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.146240    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.146253    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.146266    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:55:41.068762    5250 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc/volumes/kubernetes.io~projected/istio-token
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.146334    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token" pod "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" (UID: "4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.143405    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:41.147386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.085342    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 05:55:41.148603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.148566    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:55:41.149756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.149734    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwzt\" (UniqueName: \"kubernetes.io/projected/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-kube-api-access-kwwzt\") pod \"payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk\" (UID: \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:41.216636 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.215426    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:55:41.229476 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.229446    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"]
Oct 30 05:55:41.243140 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.243051    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:41.251202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.249581    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:55:41.254647 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.254620    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"]
Oct 30 05:55:41.288630 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.288601    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-3abe42\""
Oct 30 05:55:41.299215 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.289082    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:41.317982 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.317958    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-3abe42-dockercfg-qpv4x\""
Oct 30 05:55:41.330579 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.330551    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:41.437225 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.437161    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:41.437705 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.437668    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a746e-35d3-408d-b9c1-66890f8ad655-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc\" (UID: \"ac2a746e-35d3-408d-b9c1-66890f8ad655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:41.437837 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.437733    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgcxz\" (UniqueName: \"kubernetes.io/projected/ac2a746e-35d3-408d-b9c1-66890f8ad655-kube-api-access-zgcxz\") pod \"payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc\" (UID: \"ac2a746e-35d3-408d-b9c1-66890f8ad655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:41.462786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.462754    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"]
Oct 30 05:55:41.463079 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.463065    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"]
Oct 30 05:55:41.501406 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.501288    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:41.540617 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.540576    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b795w\" (UniqueName: \"kubernetes.io/projected/b3352c23-ff38-4ef7-8995-4db145aed854-kube-api-access-b795w\") pod \"payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv\" (UID: \"b3352c23-ff38-4ef7-8995-4db145aed854\") " pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:41.547284 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.547256    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3352c23-ff38-4ef7-8995-4db145aed854-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv\" (UID: \"b3352c23-ff38-4ef7-8995-4db145aed854\") " pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:41.547634 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.547613    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a746e-35d3-408d-b9c1-66890f8ad655-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc\" (UID: \"ac2a746e-35d3-408d-b9c1-66890f8ad655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:41.547790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.547776    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgcxz\" (UniqueName: \"kubernetes.io/projected/ac2a746e-35d3-408d-b9c1-66890f8ad655-kube-api-access-zgcxz\") pod \"payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc\" (UID: \"ac2a746e-35d3-408d-b9c1-66890f8ad655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:41.641889 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.641830    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a746e-35d3-408d-b9c1-66890f8ad655-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc\" (UID: \"ac2a746e-35d3-408d-b9c1-66890f8ad655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:41.719664 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.701406    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b795w\" (UniqueName: \"kubernetes.io/projected/b3352c23-ff38-4ef7-8995-4db145aed854-kube-api-access-b795w\") pod \"payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv\" (UID: \"b3352c23-ff38-4ef7-8995-4db145aed854\") " pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:41.719664 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.717026    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3352c23-ff38-4ef7-8995-4db145aed854-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv\" (UID: \"b3352c23-ff38-4ef7-8995-4db145aed854\") " pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:41.719664 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.718510    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3352c23-ff38-4ef7-8995-4db145aed854-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv\" (UID: \"b3352c23-ff38-4ef7-8995-4db145aed854\") " pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:41.746462 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.746423    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 05:55:41.746664 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.746501    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 05:55:41.748763 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.748738    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 05:55:41.748945 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.748924    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 05:55:41.821224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.801633    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b795w\" (UniqueName: \"kubernetes.io/projected/b3352c23-ff38-4ef7-8995-4db145aed854-kube-api-access-b795w\") pod \"payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv\" (UID: \"b3352c23-ff38-4ef7-8995-4db145aed854\") " pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:41.862204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.843944    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgcxz\" (UniqueName: \"kubernetes.io/projected/ac2a746e-35d3-408d-b9c1-66890f8ad655-kube-api-access-zgcxz\") pod \"payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc\" (UID: \"ac2a746e-35d3-408d-b9c1-66890f8ad655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:41.862204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.860486    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:41.911749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.868366    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"]
Oct 30 05:55:41.943385 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.943349    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:41.976656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:41.950864    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-4d9bq"]
Oct 30 05:55:42.523122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:42.522844    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:42.535525 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:42.535493    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"]
Oct 30 05:55:42.679032 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:42.678633    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:42.737608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:42.724554    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 05:55:42.741483 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:42.741445    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 05:55:42.778207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:42.748644    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 05:55:42.778207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:42.748697    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 05:55:43.053352 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.051669    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:55:43.274272 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.228333    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 05:55:43.512205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.484350    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 05:55:43.540415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.538266    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:55:43.553081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.553054    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc" path="/var/lib/kubelet/pods/4b4d8c64-2eb5-4cca-bfa7-7edc6a0954dc/volumes"
Oct 30 05:55:43.554278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.554254    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk" event={"ID":"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848","Type":"ContainerStarted","Data":"622497c692bcdcc22f5e9dfccdfebac98f6c2575440e4ae09799bb8dfe338894"}
Oct 30 05:55:43.721430 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.721395    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 05:55:43.721616 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.721457    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 05:55:43.749345 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.749309    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 05:55:43.749796 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.749370    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 05:55:43.821613 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.821574    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"]
Oct 30 05:55:43.918385 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:43.918343    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"]
Oct 30 05:55:43.920195 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:55:43.920156    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2a746e_35d3_408d_b9c1_66890f8ad655.slice/crio-2f9ac27b935ea9fab96bc02f4b2e299f12ff33b981a48817a65c92dc2b8612b9 WatchSource:0}: Error finding container 2f9ac27b935ea9fab96bc02f4b2e299f12ff33b981a48817a65c92dc2b8612b9: Status 404 returned error can't find the container with id 2f9ac27b935ea9fab96bc02f4b2e299f12ff33b981a48817a65c92dc2b8612b9
Oct 30 05:55:44.612769 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.612372    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk" event={"ID":"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848","Type":"ContainerStarted","Data":"3faf5a0a8304c2096b088628d0e5bf4cc3b4b94218e5ed61aa8ee44dfb608a4b"}
Oct 30 05:55:44.614746 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.614710    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" event={"ID":"ac2a746e-35d3-408d-b9c1-66890f8ad655","Type":"ContainerStarted","Data":"1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc"}
Oct 30 05:55:44.614909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.614799    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" event={"ID":"ac2a746e-35d3-408d-b9c1-66890f8ad655","Type":"ContainerStarted","Data":"2f9ac27b935ea9fab96bc02f4b2e299f12ff33b981a48817a65c92dc2b8612b9"}
Oct 30 05:55:44.614909 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.614831    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:44.616672 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.616646    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv" event={"ID":"b3352c23-ff38-4ef7-8995-4db145aed854","Type":"ContainerStarted","Data":"2b1cb8d3e31b19e77ca8e55d414db5ac18f8b3538910e209de8615bf199f09ca"}
Oct 30 05:55:44.616791 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.616689    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv" event={"ID":"b3352c23-ff38-4ef7-8995-4db145aed854","Type":"ContainerStarted","Data":"279c771af9f00c89c0a10653b082b0d52ca90a09ad8ea4889c87f80c9ac23db5"}
Oct 30 05:55:44.616791 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.616719    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:44.676204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.673000    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk" podStartSLOduration=5.672979494 podStartE2EDuration="5.672979494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:44.655106901 +0000 UTC m=+1459.486144145" watchObservedRunningTime="2026-10-30 05:55:44.672979494 +0000 UTC m=+1459.504016712"
Oct 30 05:55:44.714335 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.714227    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" podStartSLOduration=7.714209292 podStartE2EDuration="7.714209292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:44.699330036 +0000 UTC m=+1459.530367253" watchObservedRunningTime="2026-10-30 05:55:44.714209292 +0000 UTC m=+1459.545246564"
Oct 30 05:55:44.720960 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.720932    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 05:55:44.721100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.720980    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 05:55:44.744482 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.744435    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv" podStartSLOduration=5.744423282 podStartE2EDuration="5.744423282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:44.74273333 +0000 UTC m=+1459.573770551" watchObservedRunningTime="2026-10-30 05:55:44.744423282 +0000 UTC m=+1459.575460535"
Oct 30 05:55:44.748748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.748719    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 05:55:44.748866 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:44.748759    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 05:55:45.625787 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.625756    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:45.725630 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.725604    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:45.726150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.726134    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:45.726798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.726774    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:55:45.753118 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.753095    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:45.753427 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.753409    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:45.754064 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.754044    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:55:45.792640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.792611    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:45.947479 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:45.947433    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:48.444982 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:48.444937    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:48.602547 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:48.602512    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:49.342006 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:49.341969    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:51.160560 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:51.160528    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:51.654765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:51.654730    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:51.799310 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:51.799276    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:52.954400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:52.954363    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:54.956649 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:54.956612    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:55.628546 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:55.628517    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:55:55.628830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:55.628806    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 05:55:56.636409 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:56.636380    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:55:56.951110 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:56.951020    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:58.581150 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:55:58.581104    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:00.036236 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:00.036197    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:01.896278 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:01.896239    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:02.375946 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:56:02.375867    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:56:03.310878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:03.310848    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:03.457380 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:03.457277    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:05.196477 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:05.196437    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:06.880853 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:06.880816    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:08.812205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:08.811753    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:10.563203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:10.561663    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:12.436214 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:12.432876    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:12.591126 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:12.591084    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:14.246494 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:14.246457    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:14.425463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:14.425427    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:16.068314 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:16.068269    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:18.034482 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:18.034453    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:18.176061 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:18.176025    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:20.679649 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:20.679615    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:22.360898 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.360859    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"]
Oct 30 05:56:22.361481 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.361099    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm" podUID="beb1ac85-0b11-48aa-adec-c2b4498ef693" containerName="maas-api" containerID="cri-o://436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6" gracePeriod=30
Oct 30 05:56:22.452772 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.452735    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:22.798861 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.798835    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:56:22.862127 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.860474    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb1ac85-0b11-48aa-adec-c2b4498ef693-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1ac85-0b11-48aa-adec-c2b4498ef693-openshift-service-ca\") pod \"beb1ac85-0b11-48aa-adec-c2b4498ef693\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") "
Oct 30 05:56:22.862127 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.860734    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls\") pod \"beb1ac85-0b11-48aa-adec-c2b4498ef693\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") "
Oct 30 05:56:22.862127 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.860837    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-metrics-certs\") pod \"beb1ac85-0b11-48aa-adec-c2b4498ef693\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") "
Oct 30 05:56:22.862127 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.860917    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb1ac85-0b11-48aa-adec-c2b4498ef693-kube-api-access-4w79c\" (UniqueName: \"kubernetes.io/projected/beb1ac85-0b11-48aa-adec-c2b4498ef693-kube-api-access-4w79c\") pod \"beb1ac85-0b11-48aa-adec-c2b4498ef693\" (UID: \"beb1ac85-0b11-48aa-adec-c2b4498ef693\") "
Oct 30 05:56:22.878558 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.878504    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb1ac85-0b11-48aa-adec-c2b4498ef693-openshift-service-ca" pod "beb1ac85-0b11-48aa-adec-c2b4498ef693" (UID: "beb1ac85-0b11-48aa-adec-c2b4498ef693"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:56:22.881408 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.881223    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb1ac85-0b11-48aa-adec-c2b4498ef693-kube-api-access-4w79c" pod "beb1ac85-0b11-48aa-adec-c2b4498ef693" (UID: "beb1ac85-0b11-48aa-adec-c2b4498ef693"). InnerVolumeSpecName "kube-api-access-4w79c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:56:22.883246 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.882813    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"]
Oct 30 05:56:22.886653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.884260    5250 cpu_manager.go:414] "containerMap: removing container" podUID="beb1ac85-0b11-48aa-adec-c2b4498ef693" containerName="maas-api"
Oct 30 05:56:22.886653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.884282    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb1ac85-0b11-48aa-adec-c2b4498ef693" containerName="maas-api"
Oct 30 05:56:22.886653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.884495    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb1ac85-0b11-48aa-adec-c2b4498ef693" containerName="maas-api"
Oct 30 05:56:22.886653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.884558    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-metrics-certs" pod "beb1ac85-0b11-48aa-adec-c2b4498ef693" (UID: "beb1ac85-0b11-48aa-adec-c2b4498ef693"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:56:22.886653 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.884691    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls" pod "beb1ac85-0b11-48aa-adec-c2b4498ef693" (UID: "beb1ac85-0b11-48aa-adec-c2b4498ef693"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:56:22.890256 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.890110    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.937993 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.933311    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"]
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963159    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963257    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963356    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963437    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963606    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963668    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963701    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963736    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae476bab-f205-47f8-80e2-8d04be4e88e1-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.963793    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n77qf\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-kube-api-access-n77qf\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.964005    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.964025    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w79c\" (UniqueName: \"kubernetes.io/projected/beb1ac85-0b11-48aa-adec-c2b4498ef693-kube-api-access-4w79c\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.964039    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1ac85-0b11-48aa-adec-c2b4498ef693-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:22.964234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:22.964053    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb1ac85-0b11-48aa-adec-c2b4498ef693-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:23.065287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.065204    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.065287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.065273    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.065613 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.065337    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.065613 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.065522    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.066239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.066216    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.066320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.065588    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.066360 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.066342    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.066413 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.066399    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae476bab-f205-47f8-80e2-8d04be4e88e1-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.066491 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.066477    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n77qf\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-kube-api-access-n77qf\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.067121 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.066871    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.070511 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.067436    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.070511 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.070408    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.071399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.071346    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae476bab-f205-47f8-80e2-8d04be4e88e1-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.071693 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.071664    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.071881 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.071870    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.075794 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.075760    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.095707 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.095680    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n77qf\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-kube-api-access-n77qf\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.095872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.095854    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.105959 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.105934    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:23.108603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.108583    5250 generic.go:356] "Generic (PLEG): container finished" podID="beb1ac85-0b11-48aa-adec-c2b4498ef693" containerID="436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6" exitCode=0
Oct 30 05:56:23.108731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.108627    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"
Oct 30 05:56:23.108731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.108637    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm" event={"ID":"beb1ac85-0b11-48aa-adec-c2b4498ef693","Type":"ContainerDied","Data":"436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6"}
Oct 30 05:56:23.108731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.108678    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm" event={"ID":"beb1ac85-0b11-48aa-adec-c2b4498ef693","Type":"ContainerDied","Data":"f221e2422748b8c2881fa796b6bc555966cc84b878231dc99f799d1ab5e96dd4"}
Oct 30 05:56:23.108731 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.108698    5250 scope.go:122] "RemoveContainer" containerID="436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6"
Oct 30 05:56:23.127313 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.127281    5250 scope.go:122] "RemoveContainer" containerID="436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6"
Oct 30 05:56:23.127732 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:56:23.127705    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6\": container with ID starting with 436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6 not found: ID does not exist" containerID="436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6"
Oct 30 05:56:23.127839 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.127741    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6"} err="failed to get container status \"436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6\": rpc error: code = NotFound desc = could not find container \"436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6\": container with ID starting with 436b82e712fac87ca34696cdb2ed879d2d89b212ad66d2443af8f9097ce082c6 not found: ID does not exist"
Oct 30 05:56:23.170402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.170354    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"]
Oct 30 05:56:23.197582 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.176165    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-zj6cm"]
Oct 30 05:56:23.277302 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.277269    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:23.315448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.292863    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb1ac85-0b11-48aa-adec-c2b4498ef693" path="/var/lib/kubelet/pods/beb1ac85-0b11-48aa-adec-c2b4498ef693/volumes"
Oct 30 05:56:23.489096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.446757    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:23.906985 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:56:23.906955    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae476bab_f205_47f8_80e2_8d04be4e88e1.slice/crio-319c3414b55be7fdccb887df0c7170dd1bccc84a72cfc84a002670ff01a524d6 WatchSource:0}: Error finding container 319c3414b55be7fdccb887df0c7170dd1bccc84a72cfc84a002670ff01a524d6: Status 404 returned error can't find the container with id 319c3414b55be7fdccb887df0c7170dd1bccc84a72cfc84a002670ff01a524d6
Oct 30 05:56:23.907936 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.907902    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"]
Oct 30 05:56:23.910577 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.910545    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:56:23.910656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.910616    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:56:23.910656 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:23.910648    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:56:24.119867 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:24.119830    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" event={"ID":"ae476bab-f205-47f8-80e2-8d04be4e88e1","Type":"ContainerStarted","Data":"41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc"}
Oct 30 05:56:24.120075 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:24.119892    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" event={"ID":"ae476bab-f205-47f8-80e2-8d04be4e88e1","Type":"ContainerStarted","Data":"319c3414b55be7fdccb887df0c7170dd1bccc84a72cfc84a002670ff01a524d6"}
Oct 30 05:56:24.172680 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:24.172580    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" podStartSLOduration=3.172563963 podStartE2EDuration="3.172563963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:24.171142431 +0000 UTC m=+1499.002179650" watchObservedRunningTime="2026-10-30 05:56:24.172563963 +0000 UTC m=+1499.003601179"
Oct 30 05:56:24.277887 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:24.277862    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:25.083332 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:25.083296    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:25.289103 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:25.289073    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 05:56:25.289298 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:25.289133    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 05:56:25.645613 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:25.645584    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:56:25.648397 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:25.648371    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 05:56:26.280334 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:26.280269    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 05:56:26.280859 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:26.280338    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 05:56:27.248376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:27.248347    5250 scope.go:122] "RemoveContainer" containerID="c0b68b32da7be7da10bc9ea066553d60f2d0248432ec19a742981bd34c5c48b7"
Oct 30 05:56:27.288201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:27.279489    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 05:56:27.298208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:27.294298    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 05:56:27.301081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:27.301008    5250 scope.go:122] "RemoveContainer" containerID="aaf623bc9e515b97961c9c1266d7ed683ec0f9b151839d3893a80cf7e034abed"
Oct 30 05:56:27.978204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:27.977821    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:28.284706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:28.284614    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 05:56:28.284706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:28.284677    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 05:56:28.529973 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:28.529932    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:29.280207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:29.279665    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 05:56:29.280207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:29.279728    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 05:56:30.278477 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:30.278442    5250 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 05:56:30.278876 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:30.278503    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 05:56:30.817451 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:30.817413    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:31.293601 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:31.293574    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:31.294074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:31.293877    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:31.294992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:31.294972    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:56:31.476728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:31.476697    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:31.695628 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:31.695593    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:32.378986 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:56:32.378945    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:56:32.463598 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:32.463546    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:32.640708 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:32.640669    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:33.620030 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:33.619997    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:34.610390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:34.610358    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:35.434509 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:35.432890    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:35.941320 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:35.941288    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:36.461166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:36.461120    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:36.627822 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:36.627789    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:37.555108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:37.555071    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:39.313650 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:39.313613    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:41.045488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:41.045425    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:41.309398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:41.309291    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:41.484166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:41.484114    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:43.333248 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:43.333212    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:43.786882 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:43.786847    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:43.935071 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:43.935012    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:45.464139 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:45.464103    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:45.623124 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:45.623087    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:45.808259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:45.801911    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:46.049585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:46.049552    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:48.058203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:48.055632    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:48.674458 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:48.674422    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"]
Oct 30 05:56:48.674672 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:48.674651    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy" containerID="cri-o://51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df" gracePeriod=30
Oct 30 05:56:49.781817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:49.781783    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:51.650608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:51.650573    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:52.181288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:52.181251    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:52.504907 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:52.504827    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:53.783032 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:53.782904    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:54.009612 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.009588    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:56:54.083567 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.083536    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"]
Oct 30 05:56:54.084374 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.084352    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy"
Oct 30 05:56:54.084374 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.084374    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy"
Oct 30 05:56:54.084623 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.084608    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerName="istio-proxy"
Oct 30 05:56:54.088423 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.088403    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:54.107704 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.107682    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-39147c\""
Oct 30 05:56:54.109595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.109574    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"]
Oct 30 05:56:54.111109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.111086    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-39147c-dockercfg-l8xxz\""
Oct 30 05:56:54.113516 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113496    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-data\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.113648 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113631    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-credential-socket\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.113698 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113684    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-podinfo\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.113750 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113703    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-socket\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.113750 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113728    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-envoy\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.113872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113792    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-kube-api-access-lb9bz\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-kube-api-access-lb9bz\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.113872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113792    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-data" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:56:54.113978 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113908    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-certs\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.113978 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113921    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-credential-socket" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:56:54.114091 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.113983    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-token\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-token\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.114091 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.114053    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a61ae66-7883-48bb-b2d2-0c65deb92338-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a61ae66-7883-48bb-b2d2-0c65deb92338-istiod-ca-cert\") pod \"9a61ae66-7883-48bb-b2d2-0c65deb92338\" (UID: \"9a61ae66-7883-48bb-b2d2-0c65deb92338\") "
Oct 30 05:56:54.114224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.114090    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-socket" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:56:54.114754 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.114726    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5zf\" (UniqueName: \"kubernetes.io/projected/c8a35609-ba89-4d0b-9962-d464c2229e3b-kube-api-access-bv5zf\") pod \"payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk\" (UID: \"c8a35609-ba89-4d0b-9962-d464c2229e3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:54.114956 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.114932    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a35609-ba89-4d0b-9962-d464c2229e3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk\" (UID: \"c8a35609-ba89-4d0b-9962-d464c2229e3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:54.115507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.115295    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.115507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.115322    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.115507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.115337    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.115507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.115472    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-certs" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:56:54.115507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.115492    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a61ae66-7883-48bb-b2d2-0c65deb92338-istiod-ca-cert" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:56:54.116590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.116546    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-envoy" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:56:54.116964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.116942    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-kube-api-access-lb9bz" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "kube-api-access-lb9bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:56:54.118599 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.118576    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-token" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:56:54.118691 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.118596    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-podinfo" pod "9a61ae66-7883-48bb-b2d2-0c65deb92338" (UID: "9a61ae66-7883-48bb-b2d2-0c65deb92338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:56:54.216946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.216908    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5zf\" (UniqueName: \"kubernetes.io/projected/c8a35609-ba89-4d0b-9962-d464c2229e3b-kube-api-access-bv5zf\") pod \"payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk\" (UID: \"c8a35609-ba89-4d0b-9962-d464c2229e3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:54.217130 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.217053    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a35609-ba89-4d0b-9962-d464c2229e3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk\" (UID: \"c8a35609-ba89-4d0b-9962-d464c2229e3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:54.217335 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.217320    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.217384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.217341    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb9bz\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-kube-api-access-lb9bz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.217384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.217352    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a61ae66-7883-48bb-b2d2-0c65deb92338-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.217384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.217361    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.217384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.217368    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a61ae66-7883-48bb-b2d2-0c65deb92338-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.217384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.217376    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a61ae66-7883-48bb-b2d2-0c65deb92338-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:56:54.217752 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.217732    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a35609-ba89-4d0b-9962-d464c2229e3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk\" (UID: \"c8a35609-ba89-4d0b-9962-d464c2229e3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:54.240116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.240054    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5zf\" (UniqueName: \"kubernetes.io/projected/c8a35609-ba89-4d0b-9962-d464c2229e3b-kube-api-access-bv5zf\") pod \"payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk\" (UID: \"c8a35609-ba89-4d0b-9962-d464c2229e3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:54.406988 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.406959    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:54.506852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.505216    5250 generic.go:356] "Generic (PLEG): container finished" podID="9a61ae66-7883-48bb-b2d2-0c65deb92338" containerID="51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df" exitCode=0
Oct 30 05:56:54.506852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.505259    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"
Oct 30 05:56:54.506852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.505438    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" event={"ID":"9a61ae66-7883-48bb-b2d2-0c65deb92338","Type":"ContainerDied","Data":"51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df"}
Oct 30 05:56:54.506852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.505499    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn" event={"ID":"9a61ae66-7883-48bb-b2d2-0c65deb92338","Type":"ContainerDied","Data":"cd0ed9cdbf2f4e8bca4d42bea214af48315ae08a972b6c49b04cbed970526ba9"}
Oct 30 05:56:54.506852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.505532    5250 scope.go:122] "RemoveContainer" containerID="51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df"
Oct 30 05:56:54.527411 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.527394    5250 scope.go:122] "RemoveContainer" containerID="51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df"
Oct 30 05:56:54.527840 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:56:54.527822    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df\": container with ID starting with 51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df not found: ID does not exist" containerID="51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df"
Oct 30 05:56:54.527921 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.527851    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df"} err="failed to get container status \"51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df\": rpc error: code = NotFound desc = could not find container \"51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df\": container with ID starting with 51a13d2e0c1c0f723c703bfd65ba50240a62c05c442eab9fe70336d46a7453df not found: ID does not exist"
Oct 30 05:56:54.565878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.565846    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"]
Oct 30 05:56:54.577115 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.577087    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-nsrtn"]
Oct 30 05:56:54.723695 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:56:54.723652    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a35609_ba89_4d0b_9962_d464c2229e3b.slice/crio-edbf9a5c6e16c813a46c20a45e15df60aeaba237547670c1be9d8894f6ba4522 WatchSource:0}: Error finding container edbf9a5c6e16c813a46c20a45e15df60aeaba237547670c1be9d8894f6ba4522: Status 404 returned error can't find the container with id edbf9a5c6e16c813a46c20a45e15df60aeaba237547670c1be9d8894f6ba4522
Oct 30 05:56:54.724688 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:54.724658    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"]
Oct 30 05:56:55.284322 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:55.284284    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a61ae66-7883-48bb-b2d2-0c65deb92338" path="/var/lib/kubelet/pods/9a61ae66-7883-48bb-b2d2-0c65deb92338/volumes"
Oct 30 05:56:55.483775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:55.483741    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:55.518017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:55.517978    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk" event={"ID":"c8a35609-ba89-4d0b-9962-d464c2229e3b","Type":"ContainerStarted","Data":"cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7"}
Oct 30 05:56:55.518259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:55.518044    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk" event={"ID":"c8a35609-ba89-4d0b-9962-d464c2229e3b","Type":"ContainerStarted","Data":"edbf9a5c6e16c813a46c20a45e15df60aeaba237547670c1be9d8894f6ba4522"}
Oct 30 05:56:55.518259 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:55.518089    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:56:55.564718 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:55.564508    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk" podStartSLOduration=10.564494765 podStartE2EDuration="10.564494765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:55.560494749 +0000 UTC m=+1530.391531966" watchObservedRunningTime="2026-10-30 05:56:55.564494765 +0000 UTC m=+1530.395531982"
Oct 30 05:56:55.728780 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:55.728747    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:57.507831 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:57.507802    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:56:59.690500 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:56:59.690461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:02.210859 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:02.210822    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:02.409769 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:02.409730    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:57:04.949822 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:04.949789    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:06.529925 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:06.529851    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 05:57:06.900800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:06.900770    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:07.218167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.218078    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:07.601592 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.601470    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:07.659418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.659374    5250 generic.go:356] "Generic (PLEG): container finished" podID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerID="2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989" exitCode=0
Oct 30 05:57:07.659600 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.659427    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" event={"ID":"46d3b064-1ad9-48ab-b1f4-9b975b1a321a","Type":"ContainerDied","Data":"2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989"}
Oct 30 05:57:07.660024 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.660007    5250 scope.go:122] "RemoveContainer" containerID="2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989"
Oct 30 05:57:07.662110 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.662077    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:57:07.662276 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.662151    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:57:07.662276 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.662205    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:57:07.750762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:07.750686    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:08.155723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.152801    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:57:08.215206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.206926    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:08.559268 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.548804    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"]
Oct 30 05:57:08.661408 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.661373    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"]
Oct 30 05:57:08.678350 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.678265    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" event={"ID":"46d3b064-1ad9-48ab-b1f4-9b975b1a321a","Type":"ContainerStarted","Data":"9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f"}
Oct 30 05:57:08.774239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.774203    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:08.848759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.848725    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:57:08.849937 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.849908    5250 patch_prober.go:29] interesting pod/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 05:57:08.850072 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.849984    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 05:57:08.924490 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:08.924433    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:09.638531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:09.638495    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:09.852415 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:09.852380    5250 patch_prober.go:29] interesting pod/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 05:57:09.853036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:09.852994    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 05:57:10.852237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:10.849634    5250 patch_prober.go:29] interesting pod/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 05:57:10.852237 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:10.849695    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 05:57:11.849263 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:11.849164    5250 patch_prober.go:29] interesting pod/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 05:57:11.849458 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:11.849264    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 05:57:12.179286 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:12.179248    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:12.327119 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:12.327086    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:12.530852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:12.530754    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:12.848608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:12.848514    5250 patch_prober.go:29] interesting pod/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 05:57:12.848608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:12.848563    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 05:57:13.309041 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:13.309001    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:13.490444 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:13.490403    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:13.562267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:13.560291    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7" podUID="ef47781f-e949-43ab-9a47-0dadd1a25cb7" containerName="payload-processing" containerID="cri-o://bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8" gracePeriod=25
Oct 30 05:57:13.672720 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:13.672675    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh" podUID="5c4f5b67-815a-4275-b985-853a93997794" containerName="payload-pre-processing" containerID="cri-o://12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf" gracePeriod=25
Oct 30 05:57:13.919997 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:13.897260    5250 patch_prober.go:29] interesting pod/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 05:57:13.919997 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:13.897319    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 05:57:13.942882 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:13.935541    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:14.313113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.313086    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:57:14.387999 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.387906    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:57:14.403389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.399710    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"]
Oct 30 05:57:14.403389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.400892    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ef47781f-e949-43ab-9a47-0dadd1a25cb7" containerName="payload-processing"
Oct 30 05:57:14.403389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.400913    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef47781f-e949-43ab-9a47-0dadd1a25cb7" containerName="payload-processing"
Oct 30 05:57:14.403389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.400962    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5c4f5b67-815a-4275-b985-853a93997794" containerName="payload-pre-processing"
Oct 30 05:57:14.403389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.400970    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4f5b67-815a-4275-b985-853a93997794" containerName="payload-pre-processing"
Oct 30 05:57:14.403389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.401203    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef47781f-e949-43ab-9a47-0dadd1a25cb7" containerName="payload-processing"
Oct 30 05:57:14.403389 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.401228    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4f5b67-815a-4275-b985-853a93997794" containerName="payload-pre-processing"
Oct 30 05:57:14.404604 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.404584    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.410454 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.410430    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"]
Oct 30 05:57:14.419249 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.419226    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a1c17ebc-serving-cert\""
Oct 30 05:57:14.420858 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.420832    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef47781f-e949-43ab-9a47-0dadd1a25cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef47781f-e949-43ab-9a47-0dadd1a25cb7-plugins-config-volume\") pod \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\" (UID: \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\") "
Oct 30 05:57:14.421013 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.420940    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4f5b67-815a-4275-b985-853a93997794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4f5b67-815a-4275-b985-853a93997794-plugins-config-volume\") pod \"5c4f5b67-815a-4275-b985-853a93997794\" (UID: \"5c4f5b67-815a-4275-b985-853a93997794\") "
Oct 30 05:57:14.421013 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.421000    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4f5b67-815a-4275-b985-853a93997794-kube-api-access-w2f5f\" (UniqueName: \"kubernetes.io/projected/5c4f5b67-815a-4275-b985-853a93997794-kube-api-access-w2f5f\") pod \"5c4f5b67-815a-4275-b985-853a93997794\" (UID: \"5c4f5b67-815a-4275-b985-853a93997794\") "
Oct 30 05:57:14.421161 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.421053    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef47781f-e949-43ab-9a47-0dadd1a25cb7-kube-api-access-dpn8c\" (UniqueName: \"kubernetes.io/projected/ef47781f-e949-43ab-9a47-0dadd1a25cb7-kube-api-access-dpn8c\") pod \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\" (UID: \"ef47781f-e949-43ab-9a47-0dadd1a25cb7\") "
Oct 30 05:57:14.421430 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.421404    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef47781f-e949-43ab-9a47-0dadd1a25cb7-plugins-config-volume" pod "ef47781f-e949-43ab-9a47-0dadd1a25cb7" (UID: "ef47781f-e949-43ab-9a47-0dadd1a25cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:57:14.421847 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.421825    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4f5b67-815a-4275-b985-853a93997794-plugins-config-volume" pod "5c4f5b67-815a-4275-b985-853a93997794" (UID: "5c4f5b67-815a-4275-b985-853a93997794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:57:14.422213 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.422192    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef47781f-e949-43ab-9a47-0dadd1a25cb7-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:14.422287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.422221    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4f5b67-815a-4275-b985-853a93997794-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:14.424840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.424815    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef47781f-e949-43ab-9a47-0dadd1a25cb7-kube-api-access-dpn8c" pod "ef47781f-e949-43ab-9a47-0dadd1a25cb7" (UID: "ef47781f-e949-43ab-9a47-0dadd1a25cb7"). InnerVolumeSpecName "kube-api-access-dpn8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:14.424924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.424851    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4f5b67-815a-4275-b985-853a93997794-kube-api-access-w2f5f" pod "5c4f5b67-815a-4275-b985-853a93997794" (UID: "5c4f5b67-815a-4275-b985-853a93997794"). InnerVolumeSpecName "kube-api-access-w2f5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:14.524194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.524138    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-maas-api-tls\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.524416 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.524393    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80126769-23a0-4d10-8955-bd34746b26cc-openshift-service-ca\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.524480 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.524464    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qggn\" (UniqueName: \"kubernetes.io/projected/80126769-23a0-4d10-8955-bd34746b26cc-kube-api-access-9qggn\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.524588 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.524568    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-metrics-certs\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.524857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.524841    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2f5f\" (UniqueName: \"kubernetes.io/projected/5c4f5b67-815a-4275-b985-853a93997794-kube-api-access-w2f5f\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:14.524923 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.524863    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpn8c\" (UniqueName: \"kubernetes.io/projected/ef47781f-e949-43ab-9a47-0dadd1a25cb7-kube-api-access-dpn8c\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:14.626010 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.625974    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80126769-23a0-4d10-8955-bd34746b26cc-openshift-service-ca\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.626223 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.626036    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qggn\" (UniqueName: \"kubernetes.io/projected/80126769-23a0-4d10-8955-bd34746b26cc-kube-api-access-9qggn\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.626223 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.626077    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-metrics-certs\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.626381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.626234    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-maas-api-tls\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.626924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.626892    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80126769-23a0-4d10-8955-bd34746b26cc-openshift-service-ca\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.628844 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.628819    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-maas-api-tls\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.628930 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.628819    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-metrics-certs\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.649044 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.648991    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qggn\" (UniqueName: \"kubernetes.io/projected/80126769-23a0-4d10-8955-bd34746b26cc-kube-api-access-9qggn\") pod \"maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.723449 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.723418    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:14.849398 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.849365    5250 patch_prober.go:29] interesting pod/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 05:57:14.849575 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.849423    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 05:57:14.953240 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.953133    5250 generic.go:356] "Generic (PLEG): container finished" podID="5c4f5b67-815a-4275-b985-853a93997794" containerID="12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf" exitCode=0
Oct 30 05:57:14.953240 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.953202    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"
Oct 30 05:57:14.953474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.953242    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh" event={"ID":"5c4f5b67-815a-4275-b985-853a93997794","Type":"ContainerDied","Data":"12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf"}
Oct 30 05:57:14.953474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.953308    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh" event={"ID":"5c4f5b67-815a-4275-b985-853a93997794","Type":"ContainerDied","Data":"442bc9704fc988913cb4a2bfac514ba514bc161a8405c0f2a83c2869a1c74fdf"}
Oct 30 05:57:14.953474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.953337    5250 scope.go:122] "RemoveContainer" containerID="12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf"
Oct 30 05:57:14.955043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.955020    5250 generic.go:356] "Generic (PLEG): container finished" podID="ef47781f-e949-43ab-9a47-0dadd1a25cb7" containerID="bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8" exitCode=0
Oct 30 05:57:14.955154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.955065    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"
Oct 30 05:57:14.955154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.955113    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7" event={"ID":"ef47781f-e949-43ab-9a47-0dadd1a25cb7","Type":"ContainerDied","Data":"bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8"}
Oct 30 05:57:14.955287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.955166    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7" event={"ID":"ef47781f-e949-43ab-9a47-0dadd1a25cb7","Type":"ContainerDied","Data":"48fe2d581e18952a62ca453183437ebb9e7d648eea8e130a975be873487369c3"}
Oct 30 05:57:14.991448 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.991428    5250 scope.go:122] "RemoveContainer" containerID="12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf"
Oct 30 05:57:14.991759 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:14.991739    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf\": container with ID starting with 12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf not found: ID does not exist" containerID="12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf"
Oct 30 05:57:14.991805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.991767    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf"} err="failed to get container status \"12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf\": rpc error: code = NotFound desc = could not find container \"12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf\": container with ID starting with 12d57e42d4a186558da73136d7e1adba5ef4d92a879b6fdd47f9534a6a8274bf not found: ID does not exist"
Oct 30 05:57:14.991805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:14.991784    5250 scope.go:122] "RemoveContainer" containerID="bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8"
Oct 30 05:57:15.002970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.002949    5250 scope.go:122] "RemoveContainer" containerID="bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8"
Oct 30 05:57:15.003267 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:15.003248    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8\": container with ID starting with bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8 not found: ID does not exist" containerID="bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8"
Oct 30 05:57:15.003336 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.003274    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8"} err="failed to get container status \"bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8\": rpc error: code = NotFound desc = could not find container \"bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8\": container with ID starting with bf939ac9f402f453c844b3c3c47b9c4932631ded60a1921bc188cc20b5bec4c8 not found: ID does not exist"
Oct 30 05:57:15.014257 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.014234    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:15.109627 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.108283    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"]
Oct 30 05:57:15.113788 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:57:15.113753    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80126769_23a0_4d10_8955_bd34746b26cc.slice/crio-d2a7c64e832e0c54d9ed95dc534373ed37b393be3e0a92b0a8c799d14e301688 WatchSource:0}: Error finding container d2a7c64e832e0c54d9ed95dc534373ed37b393be3e0a92b0a8c799d14e301688: Status 404 returned error can't find the container with id d2a7c64e832e0c54d9ed95dc534373ed37b393be3e0a92b0a8c799d14e301688
Oct 30 05:57:15.116363 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.116338    5250 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:57:15.118867 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.118842    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fd1c5d-7c87c66c64-fsgj7"]
Oct 30 05:57:15.127258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.127222    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"]
Oct 30 05:57:15.134919 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.134894    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fd1c5d-79f598fcff-mxhdh"]
Oct 30 05:57:15.141402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.141375    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"]
Oct 30 05:57:15.288952 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.288873    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4f5b67-815a-4275-b985-853a93997794" path="/var/lib/kubelet/pods/5c4f5b67-815a-4275-b985-853a93997794/volumes"
Oct 30 05:57:15.289378 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.289363    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef47781f-e949-43ab-9a47-0dadd1a25cb7" path="/var/lib/kubelet/pods/ef47781f-e949-43ab-9a47-0dadd1a25cb7/volumes"
Oct 30 05:57:15.599437 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.599352    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:15.744593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.744562    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"]
Oct 30 05:57:15.760200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.755077    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:15.760200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.756824    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:15.772985 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.772952    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"]
Oct 30 05:57:15.773274 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.773250    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-452a59\""
Oct 30 05:57:15.773549 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.773525    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-452a59-dockercfg-xccd5\""
Oct 30 05:57:15.888258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.875523    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf5f568-1f52-426e-94a4-aece3c315cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn\" (UID: \"fcf5f568-1f52-426e-94a4-aece3c315cda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:15.888258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.875714    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x765m\" (UniqueName: \"kubernetes.io/projected/fcf5f568-1f52-426e-94a4-aece3c315cda-kube-api-access-x765m\") pod \"payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn\" (UID: \"fcf5f568-1f52-426e-94a4-aece3c315cda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:15.888258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.884852    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:57:15.888258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.886716    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:57:15.896963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.896938    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:57:15.967887 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.967852    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv" event={"ID":"80126769-23a0-4d10-8955-bd34746b26cc","Type":"ContainerStarted","Data":"27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad"}
Oct 30 05:57:15.968173 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.968149    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:15.968406 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.968248    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv" event={"ID":"80126769-23a0-4d10-8955-bd34746b26cc","Type":"ContainerStarted","Data":"d2a7c64e832e0c54d9ed95dc534373ed37b393be3e0a92b0a8c799d14e301688"}
Oct 30 05:57:15.978064 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.977286    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf5f568-1f52-426e-94a4-aece3c315cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn\" (UID: \"fcf5f568-1f52-426e-94a4-aece3c315cda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:15.978064 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.977397    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x765m\" (UniqueName: \"kubernetes.io/projected/fcf5f568-1f52-426e-94a4-aece3c315cda-kube-api-access-x765m\") pod \"payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn\" (UID: \"fcf5f568-1f52-426e-94a4-aece3c315cda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:15.978381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:15.978306    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf5f568-1f52-426e-94a4-aece3c315cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn\" (UID: \"fcf5f568-1f52-426e-94a4-aece3c315cda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:16.028810 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:16.028760    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x765m\" (UniqueName: \"kubernetes.io/projected/fcf5f568-1f52-426e-94a4-aece3c315cda-kube-api-access-x765m\") pod \"payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn\" (UID: \"fcf5f568-1f52-426e-94a4-aece3c315cda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:16.034380 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:16.034287    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv" podStartSLOduration=4.62275974 podStartE2EDuration="5.034268446s" totalImagesPullingTime="411.508706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:16.022849608 +0000 UTC m=+1550.853886827" watchObservedRunningTime="2026-10-30 05:57:16.034268446 +0000 UTC m=+1550.865305663"
Oct 30 05:57:16.101104 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:16.098031    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:16.515961 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:57:16.515932    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf5f568_1f52_426e_94a4_aece3c315cda.slice/crio-3384f907a689deb74152afbb3643df954b8a24d82034400cf26941fce561ebdb WatchSource:0}: Error finding container 3384f907a689deb74152afbb3643df954b8a24d82034400cf26941fce561ebdb: Status 404 returned error can't find the container with id 3384f907a689deb74152afbb3643df954b8a24d82034400cf26941fce561ebdb
Oct 30 05:57:16.516349 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:16.516324    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"]
Oct 30 05:57:16.985151 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:16.985111    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn" event={"ID":"fcf5f568-1f52-426e-94a4-aece3c315cda","Type":"ContainerStarted","Data":"5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f"}
Oct 30 05:57:16.985594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:16.985219    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn" event={"ID":"fcf5f568-1f52-426e-94a4-aece3c315cda","Type":"ContainerStarted","Data":"3384f907a689deb74152afbb3643df954b8a24d82034400cf26941fce561ebdb"}
Oct 30 05:57:16.985594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:16.985378    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:17.029159 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:17.029111    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn" podStartSLOduration=2.029096918 podStartE2EDuration="2.029096918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:17.02575902 +0000 UTC m=+1551.856796242" watchObservedRunningTime="2026-10-30 05:57:17.029096918 +0000 UTC m=+1551.860134135"
Oct 30 05:57:17.162432 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:17.161699    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:19.309239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:19.309069    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:19.782684 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:19.782646    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"]
Oct 30 05:57:19.813274 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:19.813246    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"]
Oct 30 05:57:19.945987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:19.945946    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:20.103937 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:20.103845    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:21.144521 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:21.144461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:21.553065 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:21.552984    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:21.991102 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:21.991068    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:57:23.189461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:23.189418    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:24.049288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:24.049241    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:24.192970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:24.192797    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:24.783759 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:24.783708    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx" podUID="1390fad0-655d-4660-a34f-4a2977f468ef" containerName="payload-processing" containerID="cri-o://895b13eb75eda9a4e117f1fbeb85813c7f0083d049c831974086e14dbb482f91" gracePeriod=25
Oct 30 05:57:24.826966 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:24.826915    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p" podUID="1d646c66-5f08-4dcd-875d-14d997ace3c1" containerName="payload-pre-processing" containerID="cri-o://44f6817ba80abd913d5f8ea2809231ecc460b2464ab5a332c004d3f7d202c447" gracePeriod=25
Oct 30 05:57:25.169230 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.169196    5250 generic.go:356] "Generic (PLEG): container finished" podID="1d646c66-5f08-4dcd-875d-14d997ace3c1" containerID="44f6817ba80abd913d5f8ea2809231ecc460b2464ab5a332c004d3f7d202c447" exitCode=0
Oct 30 05:57:25.169343 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.169246    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p" event={"ID":"1d646c66-5f08-4dcd-875d-14d997ace3c1","Type":"ContainerDied","Data":"44f6817ba80abd913d5f8ea2809231ecc460b2464ab5a332c004d3f7d202c447"}
Oct 30 05:57:25.170980 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.170955    5250 generic.go:356] "Generic (PLEG): container finished" podID="1390fad0-655d-4660-a34f-4a2977f468ef" containerID="895b13eb75eda9a4e117f1fbeb85813c7f0083d049c831974086e14dbb482f91" exitCode=0
Oct 30 05:57:25.171103 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.171030    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx" event={"ID":"1390fad0-655d-4660-a34f-4a2977f468ef","Type":"ContainerDied","Data":"895b13eb75eda9a4e117f1fbeb85813c7f0083d049c831974086e14dbb482f91"}
Oct 30 05:57:25.197699 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.197669    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:57:25.239235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.239209    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:57:25.255574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.255542    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"]
Oct 30 05:57:25.256735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.256708    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1390fad0-655d-4660-a34f-4a2977f468ef" containerName="payload-processing"
Oct 30 05:57:25.256735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.256734    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1390fad0-655d-4660-a34f-4a2977f468ef" containerName="payload-processing"
Oct 30 05:57:25.256921 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.256753    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1d646c66-5f08-4dcd-875d-14d997ace3c1" containerName="payload-pre-processing"
Oct 30 05:57:25.256921 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.256761    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d646c66-5f08-4dcd-875d-14d997ace3c1" containerName="payload-pre-processing"
Oct 30 05:57:25.257039 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.256953    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1390fad0-655d-4660-a34f-4a2977f468ef" containerName="payload-processing"
Oct 30 05:57:25.257039 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.256967    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d646c66-5f08-4dcd-875d-14d997ace3c1" containerName="payload-pre-processing"
Oct 30 05:57:25.261407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.261385    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:25.299035 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.299009    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"]
Oct 30 05:57:25.337463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.337383    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d646c66-5f08-4dcd-875d-14d997ace3c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d646c66-5f08-4dcd-875d-14d997ace3c1-plugins-config-volume\") pod \"1d646c66-5f08-4dcd-875d-14d997ace3c1\" (UID: \"1d646c66-5f08-4dcd-875d-14d997ace3c1\") "
Oct 30 05:57:25.337614 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.337513    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1390fad0-655d-4660-a34f-4a2977f468ef-kube-api-access-g52cg\" (UniqueName: \"kubernetes.io/projected/1390fad0-655d-4660-a34f-4a2977f468ef-kube-api-access-g52cg\") pod \"1390fad0-655d-4660-a34f-4a2977f468ef\" (UID: \"1390fad0-655d-4660-a34f-4a2977f468ef\") "
Oct 30 05:57:25.337694 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.337673    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d646c66-5f08-4dcd-875d-14d997ace3c1-kube-api-access-r7d6l\" (UniqueName: \"kubernetes.io/projected/1d646c66-5f08-4dcd-875d-14d997ace3c1-kube-api-access-r7d6l\") pod \"1d646c66-5f08-4dcd-875d-14d997ace3c1\" (UID: \"1d646c66-5f08-4dcd-875d-14d997ace3c1\") "
Oct 30 05:57:25.337775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.337752    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d646c66-5f08-4dcd-875d-14d997ace3c1-plugins-config-volume" pod "1d646c66-5f08-4dcd-875d-14d997ace3c1" (UID: "1d646c66-5f08-4dcd-875d-14d997ace3c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:57:25.337840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.337799    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1390fad0-655d-4660-a34f-4a2977f468ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1390fad0-655d-4660-a34f-4a2977f468ef-plugins-config-volume\") pod \"1390fad0-655d-4660-a34f-4a2977f468ef\" (UID: \"1390fad0-655d-4660-a34f-4a2977f468ef\") "
Oct 30 05:57:25.338197 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.338157    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1390fad0-655d-4660-a34f-4a2977f468ef-plugins-config-volume" pod "1390fad0-655d-4660-a34f-4a2977f468ef" (UID: "1390fad0-655d-4660-a34f-4a2977f468ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:57:25.338468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.338237    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx\" (UID: \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:25.338468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.338334    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zp2\" (UniqueName: \"kubernetes.io/projected/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-kube-api-access-s7zp2\") pod \"payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx\" (UID: \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:25.338753 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.338720    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1390fad0-655d-4660-a34f-4a2977f468ef-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:25.338753 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.338747    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d646c66-5f08-4dcd-875d-14d997ace3c1-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:25.340036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.340014    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1390fad0-655d-4660-a34f-4a2977f468ef-kube-api-access-g52cg" pod "1390fad0-655d-4660-a34f-4a2977f468ef" (UID: "1390fad0-655d-4660-a34f-4a2977f468ef"). InnerVolumeSpecName "kube-api-access-g52cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:25.340542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.340524    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d646c66-5f08-4dcd-875d-14d997ace3c1-kube-api-access-r7d6l" pod "1d646c66-5f08-4dcd-875d-14d997ace3c1" (UID: "1d646c66-5f08-4dcd-875d-14d997ace3c1"). InnerVolumeSpecName "kube-api-access-r7d6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:25.439911 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.439879    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx\" (UID: \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:25.440111 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.439945    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zp2\" (UniqueName: \"kubernetes.io/projected/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-kube-api-access-s7zp2\") pod \"payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx\" (UID: \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:25.440195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.440159    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52cg\" (UniqueName: \"kubernetes.io/projected/1390fad0-655d-4660-a34f-4a2977f468ef-kube-api-access-g52cg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:25.440195 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.440172    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7d6l\" (UniqueName: \"kubernetes.io/projected/1d646c66-5f08-4dcd-875d-14d997ace3c1-kube-api-access-r7d6l\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:25.440609 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.440588    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx\" (UID: \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:25.506397 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.506352    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zp2\" (UniqueName: \"kubernetes.io/projected/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-kube-api-access-s7zp2\") pod \"payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx\" (UID: \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:25.512079 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.512022    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:25.588072 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.587987    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:25.906247 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:57:25.906213    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3c1d2d_1c08_4ea9_b74c_426e8ab1238f.slice/crio-142bf9b30e908ad1bb1943d5ceaee3d34265e9911380118dc0a4067ef6e89dff WatchSource:0}: Error finding container 142bf9b30e908ad1bb1943d5ceaee3d34265e9911380118dc0a4067ef6e89dff: Status 404 returned error can't find the container with id 142bf9b30e908ad1bb1943d5ceaee3d34265e9911380118dc0a4067ef6e89dff
Oct 30 05:57:25.907773 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:25.907735    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"]
Oct 30 05:57:26.181040 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.180939    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx" event={"ID":"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f","Type":"ContainerStarted","Data":"12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5"}
Oct 30 05:57:26.181040 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.181013    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:26.181040 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.181036    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx" event={"ID":"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f","Type":"ContainerStarted","Data":"142bf9b30e908ad1bb1943d5ceaee3d34265e9911380118dc0a4067ef6e89dff"}
Oct 30 05:57:26.182400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.182377    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"
Oct 30 05:57:26.182547 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.182398    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p" event={"ID":"1d646c66-5f08-4dcd-875d-14d997ace3c1","Type":"ContainerDied","Data":"f5ed72fc78fc0b47efd1f91aceb2b5a495dfa3b0999128d6e2cbb87be42062af"}
Oct 30 05:57:26.182547 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.182460    5250 scope.go:122] "RemoveContainer" containerID="44f6817ba80abd913d5f8ea2809231ecc460b2464ab5a332c004d3f7d202c447"
Oct 30 05:57:26.183920 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.183899    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"
Oct 30 05:57:26.183920 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.183908    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx" event={"ID":"1390fad0-655d-4660-a34f-4a2977f468ef","Type":"ContainerDied","Data":"c5cb683a3dd64eb5e3cb2be25d8e513c0d08528aa06be8979a6161f87bbb9003"}
Oct 30 05:57:26.195077 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.195053    5250 scope.go:122] "RemoveContainer" containerID="895b13eb75eda9a4e117f1fbeb85813c7f0083d049c831974086e14dbb482f91"
Oct 30 05:57:26.222123 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.222076    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx" podStartSLOduration=11.222062193 podStartE2EDuration="11.222062193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:26.220239992 +0000 UTC m=+1561.051277209" watchObservedRunningTime="2026-10-30 05:57:26.222062193 +0000 UTC m=+1561.053099411"
Oct 30 05:57:26.259800 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.259755    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"]
Oct 30 05:57:26.264973 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.264946    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df544d-77949d54b4-stbxx"]
Oct 30 05:57:26.295293 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.295263    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"]
Oct 30 05:57:26.299444 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:26.299418    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df544d-7fd7bc8d67-js98p"]
Oct 30 05:57:27.142253 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:27.142150    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:27.287723 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:27.287683    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1390fad0-655d-4660-a34f-4a2977f468ef" path="/var/lib/kubelet/pods/1390fad0-655d-4660-a34f-4a2977f468ef/volumes"
Oct 30 05:57:27.288425 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:27.288400    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d646c66-5f08-4dcd-875d-14d997ace3c1" path="/var/lib/kubelet/pods/1d646c66-5f08-4dcd-875d-14d997ace3c1/volumes"
Oct 30 05:57:27.352782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:27.352749    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:27.353985 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:27.353965    5250 scope.go:122] "RemoveContainer" containerID="357505cfcf2bdb8a8ab9bffece83c2c4a47229e0037689f1c1d9fe80287f1285"
Oct 30 05:57:27.545101 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:27.545070    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:27.996456 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:27.996427    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 05:57:29.168790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:29.168748    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:30.934635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:30.934598    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:32.376506 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:32.376469    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:57:32.587066 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:32.587032    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:33.706550 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:33.706488    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:34.174252 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.174213    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:34.411306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.411269    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"]
Oct 30 05:57:34.411492 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.411476    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy" containerID="cri-o://f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21" gracePeriod=30
Oct 30 05:57:34.422805 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.422776    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"]
Oct 30 05:57:34.423029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.423007    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy" containerID="cri-o://14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0" gracePeriod=30
Oct 30 05:57:34.427762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.427697    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"]
Oct 30 05:57:34.427926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.427891    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy" containerID="cri-o://bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4" gracePeriod=30
Oct 30 05:57:34.435866 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.435834    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"]
Oct 30 05:57:34.436113 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:34.436086    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy" containerID="cri-o://2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2" gracePeriod=30
Oct 30 05:57:36.065572 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:36.065515    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:37.207715 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:37.207689    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 05:57:37.700431 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:37.700391    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:39.369932 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.369897    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:39.796683 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.796658    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:57:39.807460 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.807432    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:57:39.812518 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.812489    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:57:39.877031 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.876952    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"]
Oct 30 05:57:39.877760 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.877739    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy"
Oct 30 05:57:39.877760 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.877761    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy"
Oct 30 05:57:39.877931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.877820    5250 cpu_manager.go:414] "containerMap: removing container" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy"
Oct 30 05:57:39.877931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.877829    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy"
Oct 30 05:57:39.877931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.877845    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy"
Oct 30 05:57:39.877931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.877852    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy"
Oct 30 05:57:39.878138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.878029    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerName="istio-proxy"
Oct 30 05:57:39.878138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.878042    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerName="istio-proxy"
Oct 30 05:57:39.878138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.878063    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" containerName="istio-proxy"
Oct 30 05:57:39.882594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.882573    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:39.892993 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.892970    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:57:39.908492 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.908468    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"]
Oct 30 05:57:39.923966 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.923936    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-token\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-token\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.924216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.924170    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-data\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.924366 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.924349    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istiod-ca-cert\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.924511 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.924494    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-credential-socket\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.925136 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.925113    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-socket\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.925501 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.925462    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-credential-socket\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.925652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.925635    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-certs\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.925842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.925825    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-kube-api-access-g687b\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-kube-api-access-g687b\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.925976 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.925961    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istiod-ca-cert\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.926093 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.926079    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-kube-api-access-mb8gq\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-kube-api-access-mb8gq\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.926270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.926255    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-podinfo\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.927024 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.926962    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-certs" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.927307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.927285    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istiod-ca-cert" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:57:39.927773 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.927753    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istiod-ca-cert" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:57:39.928103 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.928075    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-data" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.928288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.928265    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-credential-socket" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.928416 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.928307    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-socket" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.928512 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.928414    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e16313d3-95c4-4f28-afb9-d577320359e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16313d3-95c4-4f28-afb9-d577320359e7-istio-podinfo\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.929031 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.929009    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-socket\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.929262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.929241    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-credential-socket" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.939756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.939615    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-podinfo\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.940107 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.940084    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-kube-api-access-9zzc7\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-kube-api-access-9zzc7\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.940527 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.940411    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-envoy\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.940730 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.940711    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-socket\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.941073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.941053    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-certs\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.942109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.942086    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-certs" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.943994 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.943510    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-39147c-serving-cert\""
Oct 30 05:57:39.944382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.944119    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-kube-api-access-g687b" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "kube-api-access-g687b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:39.944382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.944348    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-socket" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.944546 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.944473    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-token" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:39.944546 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.944523    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-kube-api-access-mb8gq" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "kube-api-access-mb8gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:39.944887 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.944859    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-istio-token\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-istio-token\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.945217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.944925    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-credential-socket\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.945217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.944977    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-token\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-token\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.945217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.945025    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-certs\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.945217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.945071    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-data\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.945217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.945107    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16313d3-95c4-4f28-afb9-d577320359e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16313d3-95c4-4f28-afb9-d577320359e7-istiod-ca-cert\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.945217 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.945152    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-envoy\") pod \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\" (UID: \"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931\") "
Oct 30 05:57:39.945553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.945252    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-data\") pod \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\" (UID: \"de63ae29-2a4a-44b4-b0fc-63201d94bd87\") "
Oct 30 05:57:39.945553 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.945326    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-envoy\") pod \"e16313d3-95c4-4f28-afb9-d577320359e7\" (UID: \"e16313d3-95c4-4f28-afb9-d577320359e7\") "
Oct 30 05:57:39.951243 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.949840    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-certs" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.953295 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.953266    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.953418 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.953359    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.953488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.953424    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.953659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.953641    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.954254 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.954235    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.954384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.954371    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.954530 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.954516    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.954652 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.954639    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.954748 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.954736    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.954865 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.954853    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.955014 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.954990    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g687b\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-kube-api-access-g687b\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.955142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.955028    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.955142 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.955064    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb8gq\" (UniqueName: \"kubernetes.io/projected/de63ae29-2a4a-44b4-b0fc-63201d94bd87-kube-api-access-mb8gq\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:39.955340 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.955321    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-data" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.955629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.955610    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16313d3-95c4-4f28-afb9-d577320359e7-istiod-ca-cert" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:57:39.956973 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.956943    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-podinfo" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:57:39.958222 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.958144    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-socket" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.961081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.960431    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-kube-api-access-9zzc7" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "kube-api-access-9zzc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:39.961081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.960515    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e16313d3-95c4-4f28-afb9-d577320359e7-istio-podinfo" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:57:39.961081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.960762    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-podinfo" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:57:39.961649 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.961546    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-data" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.962662 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.962632    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-credential-socket" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.962782 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.962723    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-envoy" pod "de63ae29-2a4a-44b4-b0fc-63201d94bd87" (UID: "de63ae29-2a4a-44b4-b0fc-63201d94bd87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.963072 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.963048    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-envoy" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.971836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.971811    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-istio-token" pod "e16313d3-95c4-4f28-afb9-d577320359e7" (UID: "e16313d3-95c4-4f28-afb9-d577320359e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:39.973518 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.973460    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-envoy" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:39.977837 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:39.977809    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-token" pod "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" (UID: "3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:40.056629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.056596    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-podinfo\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.056760 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.056687    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-socket\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.056760 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.056747    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-data\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.056850 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.056782    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-kube-api-access-4lrxm\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-kube-api-access-4lrxm\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.056850 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.056820    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-token\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-token\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.056946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.056893    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-envoy\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.056946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.056937    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61de2173-e6ea-4c2c-b3f5-c48200fef197-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61de2173-e6ea-4c2c-b3f5-c48200fef197-istiod-ca-cert\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.057105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.057086    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-certs\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.057526 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.057355    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-credential-socket\") pod \"61de2173-e6ea-4c2c-b3f5-c48200fef197\" (UID: \"61de2173-e6ea-4c2c-b3f5-c48200fef197\") "
Oct 30 05:57:40.057990 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.057248    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-data" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:40.058073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.057255    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-socket" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:40.058073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.057732    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-credential-socket" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:40.058073 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.057952    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61de2173-e6ea-4c2c-b3f5-c48200fef197-istiod-ca-cert" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:57:40.058282 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.058256    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-certs" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:40.058339 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.057972    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-maas-api-tls\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.058443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.058424    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-metrics-certs\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.058516 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.058488    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qbxr\" (UniqueName: \"kubernetes.io/projected/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-kube-api-access-4qbxr\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.059029 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.058813    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.059138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059046    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16313d3-95c4-4f28-afb9-d577320359e7-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059067    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61de2173-e6ea-4c2c-b3f5-c48200fef197-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059081    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059093    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059106    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059119    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059132    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16313d3-95c4-4f28-afb9-d577320359e7-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059145    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059159    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zzc7\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-kube-api-access-9zzc7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059173    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059235    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059249    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059262    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16313d3-95c4-4f28-afb9-d577320359e7-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059274    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059287    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059300    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059311    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16313d3-95c4-4f28-afb9-d577320359e7-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059325    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.059585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.059338    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de63ae29-2a4a-44b4-b0fc-63201d94bd87-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.060437 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.060414    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-envoy" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:57:40.060528 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.060432    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-podinfo" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:57:40.060594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.060549    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-kube-api-access-4lrxm" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "kube-api-access-4lrxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:40.060893 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.060871    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-token" pod "61de2173-e6ea-4c2c-b3f5-c48200fef197" (UID: "61de2173-e6ea-4c2c-b3f5-c48200fef197"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:57:40.167535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.166303    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.167535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.166532    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-maas-api-tls\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.167535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.166613    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-metrics-certs\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.167535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.166663    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qbxr\" (UniqueName: \"kubernetes.io/projected/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-kube-api-access-4qbxr\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.167535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.166774    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.167535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.166791    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrxm\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-kube-api-access-4lrxm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.167535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.166804    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.167535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.166819    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61de2173-e6ea-4c2c-b3f5-c48200fef197-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:57:40.168839 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.168791    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.171908 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.171879    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-maas-api-tls\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.172416 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.172397    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-metrics-certs\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.189498 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.189477    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qbxr\" (UniqueName: \"kubernetes.io/projected/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-kube-api-access-4qbxr\") pod \"maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.215741 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.215721    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:40.325728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.322933    5250 generic.go:356] "Generic (PLEG): container finished" podID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerID="2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2" exitCode=0
Oct 30 05:57:40.325728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.323131    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" event={"ID":"61de2173-e6ea-4c2c-b3f5-c48200fef197","Type":"ContainerDied","Data":"2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2"}
Oct 30 05:57:40.325728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.323243    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm" event={"ID":"61de2173-e6ea-4c2c-b3f5-c48200fef197","Type":"ContainerDied","Data":"ef764ec7142d53f129a293c44b323dfc8d40c9a15ee67296e53e0c27b2e397ee"}
Oct 30 05:57:40.325728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.323280    5250 scope.go:122] "RemoveContainer" containerID="2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2"
Oct 30 05:57:40.325728 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.323463    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"
Oct 30 05:57:40.327903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.326280    5250 generic.go:356] "Generic (PLEG): container finished" podID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" containerID="14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0" exitCode=0
Oct 30 05:57:40.327903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.326337    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"
Oct 30 05:57:40.327903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.326363    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" event={"ID":"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931","Type":"ContainerDied","Data":"14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0"}
Oct 30 05:57:40.327903 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.326401    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95" event={"ID":"3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931","Type":"ContainerDied","Data":"88eedb0a8e0232fed7aa66175209654f45503e5809ee271ca22b65071ac8465b"}
Oct 30 05:57:40.328267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.327975    5250 generic.go:356] "Generic (PLEG): container finished" podID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" containerID="bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4" exitCode=0
Oct 30 05:57:40.328267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.328086    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" event={"ID":"de63ae29-2a4a-44b4-b0fc-63201d94bd87","Type":"ContainerDied","Data":"bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4"}
Oct 30 05:57:40.328267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.328126    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst" event={"ID":"de63ae29-2a4a-44b4-b0fc-63201d94bd87","Type":"ContainerDied","Data":"b0c2acefbf6cd8b453e4d40abd2927fdf48350555bc0b47dab2d0ccd167b9761"}
Oct 30 05:57:40.328442 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.328266    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"
Oct 30 05:57:40.344706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.342305    5250 generic.go:356] "Generic (PLEG): container finished" podID="e16313d3-95c4-4f28-afb9-d577320359e7" containerID="f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21" exitCode=0
Oct 30 05:57:40.344706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.342381    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" event={"ID":"e16313d3-95c4-4f28-afb9-d577320359e7","Type":"ContainerDied","Data":"f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21"}
Oct 30 05:57:40.344706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.342445    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw" event={"ID":"e16313d3-95c4-4f28-afb9-d577320359e7","Type":"ContainerDied","Data":"88cab3a152218cba2903eec96aa48c7c09cdba50affa1852bcdabc0bf780719f"}
Oct 30 05:57:40.344706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.342554    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"
Oct 30 05:57:40.454409 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.452627    5250 scope.go:122] "RemoveContainer" containerID="2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2"
Oct 30 05:57:40.465774 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:40.465733    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2\": container with ID starting with 2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2 not found: ID does not exist" containerID="2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2"
Oct 30 05:57:40.465971 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.465944    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2"} err="failed to get container status \"2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2\": rpc error: code = NotFound desc = could not find container \"2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2\": container with ID starting with 2ba61a9c122609919064185f7156868e907c4c807e045df9202130c3915474a2 not found: ID does not exist"
Oct 30 05:57:40.466095 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.466083    5250 scope.go:122] "RemoveContainer" containerID="14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0"
Oct 30 05:57:40.625583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.582000    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"]
Oct 30 05:57:40.625583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.589013    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m7f95"]
Oct 30 05:57:40.625583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.601223    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"]
Oct 30 05:57:40.625583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.613884    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-m9sqm"]
Oct 30 05:57:40.625583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.622974    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"]
Oct 30 05:57:40.630241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.630165    5250 scope.go:122] "RemoveContainer" containerID="14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0"
Oct 30 05:57:40.631032 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:40.630691    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0\": container with ID starting with 14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0 not found: ID does not exist" containerID="14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0"
Oct 30 05:57:40.631032 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.630723    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0"} err="failed to get container status \"14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0\": rpc error: code = NotFound desc = could not find container \"14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0\": container with ID starting with 14d88fdcd84a974c0f55ba464e81df311b93cc2ac9b157ca260a5c9296fefcf0 not found: ID does not exist"
Oct 30 05:57:40.631032 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.630745    5250 scope.go:122] "RemoveContainer" containerID="bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4"
Oct 30 05:57:40.641576 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:57:40.641539    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a869f1e_d0f0_469f_9f6e_6dd21c35d2d5.slice/crio-f0d8f300f14a37bddb0458c42294a3d782168ef5ab7b39ef18aa83245a6841c8 WatchSource:0}: Error finding container f0d8f300f14a37bddb0458c42294a3d782168ef5ab7b39ef18aa83245a6841c8: Status 404 returned error can't find the container with id f0d8f300f14a37bddb0458c42294a3d782168ef5ab7b39ef18aa83245a6841c8
Oct 30 05:57:40.665075 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.665054    5250 scope.go:122] "RemoveContainer" containerID="bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4"
Oct 30 05:57:40.665477 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:40.665447    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4\": container with ID starting with bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4 not found: ID does not exist" containerID="bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4"
Oct 30 05:57:40.665593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.665483    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4"} err="failed to get container status \"bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4\": rpc error: code = NotFound desc = could not find container \"bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4\": container with ID starting with bc75e6e1f3166a13fbb97bcd2864377767b3f4f7ed1f5e9b17584ec1fbabc6f4 not found: ID does not exist"
Oct 30 05:57:40.665593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.665510    5250 scope.go:122] "RemoveContainer" containerID="f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21"
Oct 30 05:57:40.667474 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.667452    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"]
Oct 30 05:57:40.672043 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.672021    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2zxst"]
Oct 30 05:57:40.677417 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.677399    5250 scope.go:122] "RemoveContainer" containerID="f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21"
Oct 30 05:57:40.677735 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:40.677716    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21\": container with ID starting with f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21 not found: ID does not exist" containerID="f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21"
Oct 30 05:57:40.677803 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.677743    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21"} err="failed to get container status \"f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21\": rpc error: code = NotFound desc = could not find container \"f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21\": container with ID starting with f190307599e8052bd25c16fc37fc4caeb0affd046dc1a408943ba7d0ef71db21 not found: ID does not exist"
Oct 30 05:57:40.714967 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.714875    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"]
Oct 30 05:57:40.728013 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:40.727982    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-8hcqw"]
Oct 30 05:57:41.284523 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.284447    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931" path="/var/lib/kubelet/pods/3a4eba3a-3cac-4cc6-8fbb-b114f1eeb931/volumes"
Oct 30 05:57:41.285067 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.285045    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" path="/var/lib/kubelet/pods/61de2173-e6ea-4c2c-b3f5-c48200fef197/volumes"
Oct 30 05:57:41.285608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.285591    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de63ae29-2a4a-44b4-b0fc-63201d94bd87" path="/var/lib/kubelet/pods/de63ae29-2a4a-44b4-b0fc-63201d94bd87/volumes"
Oct 30 05:57:41.286470 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.286452    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16313d3-95c4-4f28-afb9-d577320359e7" path="/var/lib/kubelet/pods/e16313d3-95c4-4f28-afb9-d577320359e7/volumes"
Oct 30 05:57:41.357101 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.357072    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8" event={"ID":"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5","Type":"ContainerStarted","Data":"4a9a5ac6095ca5774ff97d7d49e920d733fb46cf55150490611a3fc6f1ae5dd3"}
Oct 30 05:57:41.357270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.357118    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8" event={"ID":"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5","Type":"ContainerStarted","Data":"f0d8f300f14a37bddb0458c42294a3d782168ef5ab7b39ef18aa83245a6841c8"}
Oct 30 05:57:41.357270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.357166    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:41.413655 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.413608    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8" podStartSLOduration=15.115821907 podStartE2EDuration="15.413597111s" totalImagesPullingTime="297.775204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:41.41112161 +0000 UTC m=+1576.242158830" watchObservedRunningTime="2026-10-30 05:57:41.413597111 +0000 UTC m=+1576.244634328"
Oct 30 05:57:41.503829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:41.503794    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:43.395502 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:43.395462    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:44.943108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:44.943075    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:45.445494 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:45.445464    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:46.720260 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:46.720222    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:47.369865 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:47.369835    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 05:57:48.239048 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.239010    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"]
Oct 30 05:57:48.240017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.239985    5250 cpu_manager.go:414] "containerMap: removing container" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy"
Oct 30 05:57:48.240017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.240012    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy"
Oct 30 05:57:48.240236 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.240220    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="61de2173-e6ea-4c2c-b3f5-c48200fef197" containerName="istio-proxy"
Oct 30 05:57:48.243924 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.243908    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.266918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.266887    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"]
Oct 30 05:57:48.270661 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.270634    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-452a59-serving-cert\""
Oct 30 05:57:48.388249 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.388216    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.388431 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.388291    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.388527 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.388447    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2e54f-5ae3-4276-9a26-67874c35e80a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.388783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.388753    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcxd\" (UniqueName: \"kubernetes.io/projected/c1c2e54f-5ae3-4276-9a26-67874c35e80a-kube-api-access-brcxd\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.450551 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.450489    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:48.499288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.493401    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.499288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.493482    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.499288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.493567    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2e54f-5ae3-4276-9a26-67874c35e80a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.499288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.493619    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcxd\" (UniqueName: \"kubernetes.io/projected/c1c2e54f-5ae3-4276-9a26-67874c35e80a-kube-api-access-brcxd\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.499288 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:48.494010    5250 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-serving-cert: secret "maas-api-e2e-shared-b-w1-452a59-serving-cert" not found
Oct 30 05:57:48.499288 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:57:48.494091    5250 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls podName:c1c2e54f-5ae3-4276-9a26-67874c35e80a nodeName:}" failed. No retries permitted until 2026-10-30 05:57:48.994071151 +0000 UTC m=+1583.825108357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls") pod "maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582" (UID: "c1c2e54f-5ae3-4276-9a26-67874c35e80a") : secret "maas-api-e2e-shared-b-w1-452a59-serving-cert" not found
Oct 30 05:57:48.499288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.494896    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2e54f-5ae3-4276-9a26-67874c35e80a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.504057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.504030    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.520231 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.520194    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcxd\" (UniqueName: \"kubernetes.io/projected/c1c2e54f-5ae3-4276-9a26-67874c35e80a-kube-api-access-brcxd\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:48.604917 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.604882    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:48.904304 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:48.904261    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:49.001429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:49.001402    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:49.004042 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:49.004018    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:49.159444 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:49.159352    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:49.476607 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:49.476577    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"]
Oct 30 05:57:49.478876 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:57:49.478847    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c2e54f_5ae3_4276_9a26_67874c35e80a.slice/crio-eba316533682c6bc11eb5880760c9c0318a4c612180d39c926a139c4ec79ca96 WatchSource:0}: Error finding container eba316533682c6bc11eb5880760c9c0318a4c612180d39c926a139c4ec79ca96: Status 404 returned error can't find the container with id eba316533682c6bc11eb5880760c9c0318a4c612180d39c926a139c4ec79ca96
Oct 30 05:57:50.288544 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:50.288510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:50.494270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:50.494239    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582" event={"ID":"c1c2e54f-5ae3-4276-9a26-67874c35e80a","Type":"ContainerStarted","Data":"db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a"}
Oct 30 05:57:50.494674 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:50.494296    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582" event={"ID":"c1c2e54f-5ae3-4276-9a26-67874c35e80a","Type":"ContainerStarted","Data":"eba316533682c6bc11eb5880760c9c0318a4c612180d39c926a139c4ec79ca96"}
Oct 30 05:57:50.494674 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:50.494335    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:50.539122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:50.539024    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582" podStartSLOduration=2.24129288 podStartE2EDuration="2.539007987s" totalImagesPullingTime="297.715107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:50.538105417 +0000 UTC m=+1585.369142634" watchObservedRunningTime="2026-10-30 05:57:50.539007987 +0000 UTC m=+1585.370045205"
Oct 30 05:57:52.344920 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:52.344884    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:54.120262 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:54.120226    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:55.760640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:55.760600    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:56.030306 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.030231    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8862-c11b3c"]
Oct 30 05:57:56.034960 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.034941    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8862-c11b3c"
Oct 30 05:57:56.050437 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.050391    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8862-c11b3c"]
Oct 30 05:57:56.055875 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.055852    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bnq\""
Oct 30 05:57:56.148839 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.148800    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxbz\" (UniqueName: \"kubernetes.io/projected/c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6-kube-api-access-vkxbz\") pod \"test-curl-8862-c11b3c\" (UID: \"c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6\") " pod="opendatahub/test-curl-8862-c11b3c"
Oct 30 05:57:56.249931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.249895    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxbz\" (UniqueName: \"kubernetes.io/projected/c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6-kube-api-access-vkxbz\") pod \"test-curl-8862-c11b3c\" (UID: \"c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6\") " pod="opendatahub/test-curl-8862-c11b3c"
Oct 30 05:57:56.274236 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.274208    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxbz\" (UniqueName: \"kubernetes.io/projected/c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6-kube-api-access-vkxbz\") pod \"test-curl-8862-c11b3c\" (UID: \"c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6\") " pod="opendatahub/test-curl-8862-c11b3c"
Oct 30 05:57:56.350791 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.350708    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8862-c11b3c"
Oct 30 05:57:56.509373 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.509350    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 05:57:56.679399 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:56.679311    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8862-c11b3c"]
Oct 30 05:57:56.681493 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:57:56.681461    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3e7cdf8_bad9_4d30_bb6c_8e78133e70a6.slice/crio-477066c6cc5cf7943dcec3e959499e9cc638a6db6906c3526d2fc632d721eab2 WatchSource:0}: Error finding container 477066c6cc5cf7943dcec3e959499e9cc638a6db6906c3526d2fc632d721eab2: Status 404 returned error can't find the container with id 477066c6cc5cf7943dcec3e959499e9cc638a6db6906c3526d2fc632d721eab2
Oct 30 05:57:57.468659 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:57.468625    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:57.559799 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:57.559768    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8862-c11b3c" event={"ID":"c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6","Type":"ContainerStarted","Data":"f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3"}
Oct 30 05:57:57.559971 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:57.559809    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8862-c11b3c" event={"ID":"c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6","Type":"ContainerStarted","Data":"477066c6cc5cf7943dcec3e959499e9cc638a6db6906c3526d2fc632d721eab2"}
Oct 30 05:57:57.599709 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:57.599659    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8862-c11b3c" podStartSLOduration=1.59964241 podStartE2EDuration="1.59964241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:57.596750224 +0000 UTC m=+1592.427787441" watchObservedRunningTime="2026-10-30 05:57:57.59964241 +0000 UTC m=+1592.430679627"
Oct 30 05:57:58.631208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:58.631148    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8862-c11b3c"]
Oct 30 05:57:58.638120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:58.638088    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8862-c11b3c"]
Oct 30 05:57:59.421205 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:59.418580    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:57:59.574487 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:57:59.574451    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8862-c11b3c" podUID="c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6" containerName="test-curl-8862-c11b3c" containerID="cri-o://f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3" gracePeriod=2
Oct 30 05:58:01.280226 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:01.277822    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:01.814362 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:01.814338    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8862-c11b3c"
Oct 30 05:58:01.941572 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:01.941541    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6-kube-api-access-vkxbz\" (UniqueName: \"kubernetes.io/projected/c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6-kube-api-access-vkxbz\") pod \"c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6\" (UID: \"c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6\") "
Oct 30 05:58:01.943860 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:01.943824    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6-kube-api-access-vkxbz" pod "c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6" (UID: "c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6"). InnerVolumeSpecName "kube-api-access-vkxbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:02.044056 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:02.044025    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxbz\" (UniqueName: \"kubernetes.io/projected/c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6-kube-api-access-vkxbz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:02.376112 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:02.376028    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:58:02.607814 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:02.607776    5250 generic.go:356] "Generic (PLEG): container finished" podID="c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6" containerID="f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3" exitCode=137
Oct 30 05:58:02.607814 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:02.607818    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8862-c11b3c"
Oct 30 05:58:02.608079 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:02.607883    5250 scope.go:122] "RemoveContainer" containerID="f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3"
Oct 30 05:58:02.622260 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:02.622233    5250 scope.go:122] "RemoveContainer" containerID="f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3"
Oct 30 05:58:02.622770 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:02.622749    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3\": container with ID starting with f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3 not found: ID does not exist" containerID="f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3"
Oct 30 05:58:02.622852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:02.622783    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3"} err="failed to get container status \"f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3\": rpc error: code = NotFound desc = could not find container \"f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3\": container with ID starting with f0eff104e783094280a4d715c60d1bbeba4cf470d4572a16a43a0405031485e3 not found: ID does not exist"
Oct 30 05:58:02.629667 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:02.629611    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8862-c11b3c\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6" pod="opendatahub/test-curl-8862-c11b3c"
Oct 30 05:58:03.127139 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:03.126964    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:03.282886 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:03.282851    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6" path="/var/lib/kubelet/pods/c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6/volumes"
Oct 30 05:58:04.849405 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:04.849368    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:06.574505 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:06.574452    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:06.942641 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:06.942609    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:08.236763 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.236730    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"]
Oct 30 05:58:08.237507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.237489    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6" containerName="test-curl-8862-c11b3c"
Oct 30 05:58:08.237507 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.237508    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6" containerName="test-curl-8862-c11b3c"
Oct 30 05:58:08.237640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.237625    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e7cdf8-bad9-4d30-bb6c-8e78133e70a6" containerName="test-curl-8862-c11b3c"
Oct 30 05:58:08.242836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.242813    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.274381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.274349    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"]
Oct 30 05:58:08.278700 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.278666    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b3e625\"/\"openshift-service-ca.crt\""
Oct 30 05:58:08.278847 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.278703    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b3e625\"/\"kube-root-ca.crt\""
Oct 30 05:58:08.278847 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.278762    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b3e625\"/\"default-dockercfg-zpjzw\""
Oct 30 05:58:08.278847 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.278784    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b3e625\"/\"test-model-b3e625-kserve-self-signed-certs\""
Oct 30 05:58:08.370396 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.370355    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-home\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.370577 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.370418    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-model-cache\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.370577 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.370557    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-dshm\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.370727 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.370711    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4049cd-530d-4fa0-9f01-9671302606bf-tls-certs\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.370784 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.370748    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2dw\" (UniqueName: \"kubernetes.io/projected/4a4049cd-530d-4fa0-9f01-9671302606bf-kube-api-access-df2dw\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.370833 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.370785    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-tmp-dir\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.374042 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.374010    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:08.472680 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.472648    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-tmp-dir\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.472862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.472761    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-home\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.472862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.472808    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-model-cache\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.473001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.472904    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-dshm\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.473091 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.473071    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4049cd-530d-4fa0-9f01-9671302606bf-tls-certs\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.473162 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.473131    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2dw\" (UniqueName: \"kubernetes.io/projected/4a4049cd-530d-4fa0-9f01-9671302606bf-kube-api-access-df2dw\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.473162 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.473148    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-tmp-dir\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.473288 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.473256    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-home\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.473346 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.473285    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-model-cache\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.475554 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.475525    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-dshm\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.475798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.475782    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4049cd-530d-4fa0-9f01-9671302606bf-tls-certs\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.526407 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.526330    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2dw\" (UniqueName: \"kubernetes.io/projected/4a4049cd-530d-4fa0-9f01-9671302606bf-kube-api-access-df2dw\") pod \"test-model-b3e625-kserve-5f844f6c6f-xlm8p\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") " pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.576971 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.576873    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:08.764244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.764212    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"]
Oct 30 05:58:08.766166 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.766136    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz" podUID="5d769840-311e-429d-9aba-4533ee9f66b7" containerName="maas-api" containerID="cri-o://547e9ebf131cda429e8beea6ff89a906134b29cfc0d86b2646e69e6a887b55f8" gracePeriod=30
Oct 30 05:58:08.932610 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:08.929639    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:09.363463 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.363431    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"]
Oct 30 05:58:09.363933 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.363529    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:09.625853 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.625713    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:09.803360 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.802988    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:09.813400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.810478    5250 generic.go:356] "Generic (PLEG): container finished" podID="5d769840-311e-429d-9aba-4533ee9f66b7" containerID="547e9ebf131cda429e8beea6ff89a906134b29cfc0d86b2646e69e6a887b55f8" exitCode=0
Oct 30 05:58:09.813400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.810781    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz" event={"ID":"5d769840-311e-429d-9aba-4533ee9f66b7","Type":"ContainerDied","Data":"547e9ebf131cda429e8beea6ff89a906134b29cfc0d86b2646e69e6a887b55f8"}
Oct 30 05:58:09.813400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.810871    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz" event={"ID":"5d769840-311e-429d-9aba-4533ee9f66b7","Type":"ContainerDied","Data":"7f0d73ab11e3b259b9c2ae3545aaf324401b2ffadc14c7385ed3dee91c9f1010"}
Oct 30 05:58:09.813400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.810911    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f0d73ab11e3b259b9c2ae3545aaf324401b2ffadc14c7385ed3dee91c9f1010"
Oct 30 05:58:09.813400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.812934    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p" event={"ID":"4a4049cd-530d-4fa0-9f01-9671302606bf","Type":"ContainerStarted","Data":"3f95830ada37244e7e3450d28b41db0395359438b37cc671edba4e3c39fb6f4b"}
Oct 30 05:58:09.828251 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.828221    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:58:09.922203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.914409    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d769840-311e-429d-9aba-4533ee9f66b7-kube-api-access-2rs86\" (UniqueName: \"kubernetes.io/projected/5d769840-311e-429d-9aba-4533ee9f66b7-kube-api-access-2rs86\") pod \"5d769840-311e-429d-9aba-4533ee9f66b7\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") "
Oct 30 05:58:09.922203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.914770    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-metrics-certs\") pod \"5d769840-311e-429d-9aba-4533ee9f66b7\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") "
Oct 30 05:58:09.922203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.914828    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-maas-api-tls\") pod \"5d769840-311e-429d-9aba-4533ee9f66b7\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") "
Oct 30 05:58:09.922203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.914868    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d769840-311e-429d-9aba-4533ee9f66b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d769840-311e-429d-9aba-4533ee9f66b7-openshift-service-ca\") pod \"5d769840-311e-429d-9aba-4533ee9f66b7\" (UID: \"5d769840-311e-429d-9aba-4533ee9f66b7\") "
Oct 30 05:58:09.948200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.926369    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-maas-api-tls" pod "5d769840-311e-429d-9aba-4533ee9f66b7" (UID: "5d769840-311e-429d-9aba-4533ee9f66b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:58:09.948200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.926638    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d769840-311e-429d-9aba-4533ee9f66b7-openshift-service-ca" pod "5d769840-311e-429d-9aba-4533ee9f66b7" (UID: "5d769840-311e-429d-9aba-4533ee9f66b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:09.948200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.931747    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-metrics-certs" pod "5d769840-311e-429d-9aba-4533ee9f66b7" (UID: "5d769840-311e-429d-9aba-4533ee9f66b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:58:09.972201 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:09.968387    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d769840-311e-429d-9aba-4533ee9f66b7-kube-api-access-2rs86" pod "5d769840-311e-429d-9aba-4533ee9f66b7" (UID: "5d769840-311e-429d-9aba-4533ee9f66b7"). InnerVolumeSpecName "kube-api-access-2rs86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:10.018203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.017777    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:10.018203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.017813    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d769840-311e-429d-9aba-4533ee9f66b7-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:10.018203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.017829    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d769840-311e-429d-9aba-4533ee9f66b7-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:10.018203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.017845    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rs86\" (UniqueName: \"kubernetes.io/projected/5d769840-311e-429d-9aba-4533ee9f66b7-kube-api-access-2rs86\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:10.088258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.088141    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:10.410244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.409016    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:10.727287 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.723219    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:10.822458 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.822423    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p" event={"ID":"4a4049cd-530d-4fa0-9f01-9671302606bf","Type":"ContainerStarted","Data":"b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed"}
Oct 30 05:58:10.822615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.822483    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"
Oct 30 05:58:10.869351 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.869295    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p" podStartSLOduration=2.869276676 podStartE2EDuration="2.869276676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:10.864855504 +0000 UTC m=+1605.695892720" watchObservedRunningTime="2026-10-30 05:58:10.869276676 +0000 UTC m=+1605.700313895"
Oct 30 05:58:10.920358 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.920317    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"]
Oct 30 05:58:10.925235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:10.925196    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd1c5d-6c48dbbf95-h2qlz"]
Oct 30 05:58:11.283734 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:11.283699    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d769840-311e-429d-9aba-4533ee9f66b7" path="/var/lib/kubelet/pods/5d769840-311e-429d-9aba-4533ee9f66b7/volumes"
Oct 30 05:58:11.577349 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:11.577276    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:13.147775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:13.147742    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:15.310817 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:15.310645    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:16.940603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:16.940564    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:17.855896 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:17.855865    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:18.016273 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:18.016241    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:18.304854 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:18.304789    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:18.577505 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:18.577420    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:18.593132 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:18.593097    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:18.932819 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:18.932791    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 05:58:19.126148 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:19.126108    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:21.098070 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.098040    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"]
Oct 30 05:58:21.098597 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.098376    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6" podUID="9c1c44db-9f95-42b5-bac6-9a827a1fd006" containerName="maas-api" containerID="cri-o://21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1" gracePeriod=30
Oct 30 05:58:21.239700 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.239662    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:21.578706 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.578667    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:58:21.592585 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.592547    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-metrics-certs\") pod \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") "
Oct 30 05:58:21.593790 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.593743    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1c44db-9f95-42b5-bac6-9a827a1fd006-kube-api-access-t5974\" (UniqueName: \"kubernetes.io/projected/9c1c44db-9f95-42b5-bac6-9a827a1fd006-kube-api-access-t5974\") pod \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") "
Oct 30 05:58:21.594851 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.594832    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls\") pod \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") "
Oct 30 05:58:21.595087 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.595071    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1c44db-9f95-42b5-bac6-9a827a1fd006-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c44db-9f95-42b5-bac6-9a827a1fd006-openshift-service-ca\") pod \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\" (UID: \"9c1c44db-9f95-42b5-bac6-9a827a1fd006\") "
Oct 30 05:58:21.595708 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.595579    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1c44db-9f95-42b5-bac6-9a827a1fd006-openshift-service-ca" pod "9c1c44db-9f95-42b5-bac6-9a827a1fd006" (UID: "9c1c44db-9f95-42b5-bac6-9a827a1fd006"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:21.596765 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.596743    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c44db-9f95-42b5-bac6-9a827a1fd006-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:21.615492 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.615344    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-metrics-certs" pod "9c1c44db-9f95-42b5-bac6-9a827a1fd006" (UID: "9c1c44db-9f95-42b5-bac6-9a827a1fd006"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:58:21.616390 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.616364    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1c44db-9f95-42b5-bac6-9a827a1fd006-kube-api-access-t5974" pod "9c1c44db-9f95-42b5-bac6-9a827a1fd006" (UID: "9c1c44db-9f95-42b5-bac6-9a827a1fd006"). InnerVolumeSpecName "kube-api-access-t5974". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:21.617247 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.617211    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls" pod "9c1c44db-9f95-42b5-bac6-9a827a1fd006" (UID: "9c1c44db-9f95-42b5-bac6-9a827a1fd006"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:58:21.708775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.708733    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:21.708775 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.708774    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c44db-9f95-42b5-bac6-9a827a1fd006-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:21.709042 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.708790    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5974\" (UniqueName: \"kubernetes.io/projected/9c1c44db-9f95-42b5-bac6-9a827a1fd006-kube-api-access-t5974\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:21.974552 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.974514    5250 generic.go:356] "Generic (PLEG): container finished" podID="9c1c44db-9f95-42b5-bac6-9a827a1fd006" containerID="21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1" exitCode=0
Oct 30 05:58:21.974756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.974636    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6" event={"ID":"9c1c44db-9f95-42b5-bac6-9a827a1fd006","Type":"ContainerDied","Data":"21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1"}
Oct 30 05:58:21.974756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.974688    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6" event={"ID":"9c1c44db-9f95-42b5-bac6-9a827a1fd006","Type":"ContainerDied","Data":"a621f83534a191c0f5cddf04674d913703b5d5214269022517986f73848e47e4"}
Oct 30 05:58:21.974756 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.974715    5250 scope.go:122] "RemoveContainer" containerID="21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1"
Oct 30 05:58:21.974943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:21.974869    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"
Oct 30 05:58:22.000798 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:22.000755    5250 scope.go:122] "RemoveContainer" containerID="21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1"
Oct 30 05:58:22.001430 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:22.001400    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1\": container with ID starting with 21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1 not found: ID does not exist" containerID="21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1"
Oct 30 05:58:22.001581 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:22.001439    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1"} err="failed to get container status \"21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1\": rpc error: code = NotFound desc = could not find container \"21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1\": container with ID starting with 21ccc9a0392af82edd840f5d536e16d7dc8dabae39c9aaea9cb4e49ef13db2e1 not found: ID does not exist"
Oct 30 05:58:22.026621 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:22.026586    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:22.034480 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:22.034448    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"]
Oct 30 05:58:22.044928 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:22.044899    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df544d-66fd667b99-l6hd6"]
Oct 30 05:58:22.235582 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:22.235499    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:22.412861 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:22.412832    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:22.842466 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:22.842434    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:23.283377 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:23.283348    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1c44db-9f95-42b5-bac6-9a827a1fd006" path="/var/lib/kubelet/pods/9c1c44db-9f95-42b5-bac6-9a827a1fd006/volumes"
Oct 30 05:58:24.038067 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:24.038032    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:24.761053 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:24.761014    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:25.027899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:25.027799    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:25.579274 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:25.579238    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:25.727969 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:25.727928    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:27.859412 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:27.859313    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:29.360769 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:29.360714    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:29.519409 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:29.519370    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:29.700777 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:29.698635    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"]
Oct 30 05:58:29.700777 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:29.698933    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy" containerID="cri-o://4563e0429fd9ae42832851a6df25065881e91ac444c603d870300fc949f9833f" gracePeriod=30
Oct 30 05:58:29.705269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:29.705235    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:29.862203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:29.861689    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:31.764696 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:31.764666    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"]
Oct 30 05:58:31.765116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:31.764859    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy" containerID="cri-o://4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04" gracePeriod=30
Oct 30 05:58:31.915803 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:31.915727    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:32.376135 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:32.376101    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:58:33.098991 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.098962    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:33.402621 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.402586    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:33.702888 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.702803    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"]
Oct 30 05:58:33.703059 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.703013    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy" containerID="cri-o://41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc" gracePeriod=30
Oct 30 05:58:33.713933 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.713906    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"]
Oct 30 05:58:33.714120 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.714100    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy" containerID="cri-o://ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76" gracePeriod=30
Oct 30 05:58:33.739629 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.739593    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:33.740460 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.740433    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"]
Oct 30 05:58:33.740669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.740646    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy" containerID="cri-o://d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e" gracePeriod=30
Oct 30 05:58:33.937216 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:33.935580    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:34.295509 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:34.295474    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"]
Oct 30 05:58:34.296085 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:34.295770    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv" podUID="80126769-23a0-4d10-8955-bd34746b26cc" containerName="maas-api" containerID="cri-o://27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad" gracePeriod=30
Oct 30 05:58:34.301970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:34.301770    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:34.557454 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:34.553496    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:35.075615 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.074970    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:58:35.124204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.120700    5250 generic.go:356] "Generic (PLEG): container finished" podID="35668233-ef08-4087-a61d-f788c405e7d9" containerID="4563e0429fd9ae42832851a6df25065881e91ac444c603d870300fc949f9833f" exitCode=0
Oct 30 05:58:35.124204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.121390    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" event={"ID":"35668233-ef08-4087-a61d-f788c405e7d9","Type":"ContainerDied","Data":"4563e0429fd9ae42832851a6df25065881e91ac444c603d870300fc949f9833f"}
Oct 30 05:58:35.133258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.131827    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-maas-api-tls\") pod \"80126769-23a0-4d10-8955-bd34746b26cc\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") "
Oct 30 05:58:35.133258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.131935    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-metrics-certs\") pod \"80126769-23a0-4d10-8955-bd34746b26cc\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") "
Oct 30 05:58:35.133258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.132035    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80126769-23a0-4d10-8955-bd34746b26cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80126769-23a0-4d10-8955-bd34746b26cc-openshift-service-ca\") pod \"80126769-23a0-4d10-8955-bd34746b26cc\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") "
Oct 30 05:58:35.133258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.132152    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80126769-23a0-4d10-8955-bd34746b26cc-kube-api-access-9qggn\" (UniqueName: \"kubernetes.io/projected/80126769-23a0-4d10-8955-bd34746b26cc-kube-api-access-9qggn\") pod \"80126769-23a0-4d10-8955-bd34746b26cc\" (UID: \"80126769-23a0-4d10-8955-bd34746b26cc\") "
Oct 30 05:58:35.136766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.136592    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80126769-23a0-4d10-8955-bd34746b26cc-openshift-service-ca" pod "80126769-23a0-4d10-8955-bd34746b26cc" (UID: "80126769-23a0-4d10-8955-bd34746b26cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:35.136766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.136726    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80126769-23a0-4d10-8955-bd34746b26cc-kube-api-access-9qggn" pod "80126769-23a0-4d10-8955-bd34746b26cc" (UID: "80126769-23a0-4d10-8955-bd34746b26cc"). InnerVolumeSpecName "kube-api-access-9qggn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:35.141355 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.141307    5250 generic.go:356] "Generic (PLEG): container finished" podID="80126769-23a0-4d10-8955-bd34746b26cc" containerID="27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad" exitCode=0
Oct 30 05:58:35.141503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.141457    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv" event={"ID":"80126769-23a0-4d10-8955-bd34746b26cc","Type":"ContainerDied","Data":"27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad"}
Oct 30 05:58:35.141581 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.141508    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv" event={"ID":"80126769-23a0-4d10-8955-bd34746b26cc","Type":"ContainerDied","Data":"d2a7c64e832e0c54d9ed95dc534373ed37b393be3e0a92b0a8c799d14e301688"}
Oct 30 05:58:35.141581 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.141538    5250 scope.go:122] "RemoveContainer" containerID="27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad"
Oct 30 05:58:35.141719 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.141701    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"
Oct 30 05:58:35.142338 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.142304    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-metrics-certs" pod "80126769-23a0-4d10-8955-bd34746b26cc" (UID: "80126769-23a0-4d10-8955-bd34746b26cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:58:35.142905 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.142884    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-maas-api-tls" pod "80126769-23a0-4d10-8955-bd34746b26cc" (UID: "80126769-23a0-4d10-8955-bd34746b26cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:58:35.185618 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.184479    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:35.201830 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.201799    5250 scope.go:122] "RemoveContainer" containerID="27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad"
Oct 30 05:58:35.202324 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:35.202295    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad\": container with ID starting with 27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad not found: ID does not exist" containerID="27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad"
Oct 30 05:58:35.202437 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.202338    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad"} err="failed to get container status \"27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad\": rpc error: code = NotFound desc = could not find container \"27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad\": container with ID starting with 27d923e76702e032e4e3671a1ac4d0337437a1b5de4869b9f776308f8af450ad not found: ID does not exist"
Oct 30 05:58:35.234811 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.234779    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80126769-23a0-4d10-8955-bd34746b26cc-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.234943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.234816    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qggn\" (UniqueName: \"kubernetes.io/projected/80126769-23a0-4d10-8955-bd34746b26cc-kube-api-access-9qggn\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.234943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.234831    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.234943 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.234845    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80126769-23a0-4d10-8955-bd34746b26cc-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.338935 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.338845    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:35.408971 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.408943    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:58:35.439735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.439701    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-socket\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.439926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.439797    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-envoy\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.439926 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.439865    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-data\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.440054 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.439927    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-istio-token\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-istio-token\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.440054 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440010    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-kube-api-access-cb5hg\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-kube-api-access-cb5hg\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.440054 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440013    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-socket" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:35.440279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440132    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35668233-ef08-4087-a61d-f788c405e7d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35668233-ef08-4087-a61d-f788c405e7d9-istio-podinfo\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.440279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440206    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35668233-ef08-4087-a61d-f788c405e7d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35668233-ef08-4087-a61d-f788c405e7d9-istiod-ca-cert\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.440279 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440234    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-certs\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.440459 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440283    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-data" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:35.440459 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440343    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-credential-socket\") pod \"35668233-ef08-4087-a61d-f788c405e7d9\" (UID: \"35668233-ef08-4087-a61d-f788c405e7d9\") "
Oct 30 05:58:35.440608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440584    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-certs" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:35.441443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440821    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-credential-socket" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:35.441443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.440934    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35668233-ef08-4087-a61d-f788c405e7d9-istiod-ca-cert" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:35.441443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.441346    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.441443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.441367    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.441443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.441381    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.441443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.441394    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35668233-ef08-4087-a61d-f788c405e7d9-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.441443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.441421    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.443918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.443891    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-istio-token" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:35.445603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.445571    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-envoy" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:35.449678 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.449653    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35668233-ef08-4087-a61d-f788c405e7d9-istio-podinfo" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:58:35.451515 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.451485    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-kube-api-access-cb5hg" pod "35668233-ef08-4087-a61d-f788c405e7d9" (UID: "35668233-ef08-4087-a61d-f788c405e7d9"). InnerVolumeSpecName "kube-api-access-cb5hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:35.526640 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.526602    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:35.530327 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.530256    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"]
Oct 30 05:58:35.534754 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.534729    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1c17ebc-688b97fc96-k69jv"]
Oct 30 05:58:35.543300 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.543279    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35668233-ef08-4087-a61d-f788c405e7d9-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.543458 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.543438    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.543458 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.543459    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5hg\" (UniqueName: \"kubernetes.io/projected/35668233-ef08-4087-a61d-f788c405e7d9-kube-api-access-cb5hg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.543617 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.543471    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35668233-ef08-4087-a61d-f788c405e7d9-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:35.791247 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:35.791112    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:36.119154 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:36.119077    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:36.154859 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:36.154820    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd" event={"ID":"35668233-ef08-4087-a61d-f788c405e7d9","Type":"ContainerDied","Data":"c719da2606465e620d1b9ac4f81899d099d7e15aa48f2f0e21749dbdaa212790"}
Oct 30 05:58:36.155016 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:36.154910    5250 scope.go:122] "RemoveContainer" containerID="4563e0429fd9ae42832851a6df25065881e91ac444c603d870300fc949f9833f"
Oct 30 05:58:36.155105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:36.155077    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"
Oct 30 05:58:36.213362 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:36.213332    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"]
Oct 30 05:58:36.224724 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:36.224699    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-h8sfd"]
Oct 30 05:58:36.591592 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:36.591558    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:37.107458 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.107430    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:58:37.148902 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.148831    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:37.166005 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.165970    5250 generic.go:356] "Generic (PLEG): container finished" podID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerID="4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04" exitCode=0
Oct 30 05:58:37.166250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.166055    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"
Oct 30 05:58:37.166250 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.166141    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" event={"ID":"12fa15c4-7aa4-4664-8a37-10a6f3f2e214","Type":"ContainerDied","Data":"4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04"}
Oct 30 05:58:37.166401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.166250    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95" event={"ID":"12fa15c4-7aa4-4664-8a37-10a6f3f2e214","Type":"ContainerDied","Data":"e28551dcb2a8ee6e9b041c514a42e72a4a08cf3a0afc71b369e3f532f1f80aa2"}
Oct 30 05:58:37.166401 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.166287    5250 scope.go:122] "RemoveContainer" containerID="4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04"
Oct 30 05:58:37.175416 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.175386    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-token\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-token\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.175536 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.175452    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-socket\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.175536 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.175507    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-certs\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.175676 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.175624    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-envoy\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.175770 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.175748    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-kube-api-access-f4djs\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-kube-api-access-f4djs\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.175836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.175798    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-certs" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:37.175836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.175814    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istiod-ca-cert\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.175942 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.175905    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-credential-socket\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.176022 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.176006    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-podinfo\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.176100 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.176085    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-data\") pod \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\" (UID: \"12fa15c4-7aa4-4664-8a37-10a6f3f2e214\") "
Oct 30 05:58:37.176722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.176691    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-data" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:37.176842 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.176614    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istiod-ca-cert" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:37.177044 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.176950    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-credential-socket" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:37.177229 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.177167    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-socket" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:37.187987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.187570    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-kube-api-access-f4djs" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "kube-api-access-f4djs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:37.187987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.187671    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-envoy" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:37.187987 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.187904    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-token" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:37.190744 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.188304    5250 scope.go:122] "RemoveContainer" containerID="4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04"
Oct 30 05:58:37.190744 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.188397    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-podinfo" pod "12fa15c4-7aa4-4664-8a37-10a6f3f2e214" (UID: "12fa15c4-7aa4-4664-8a37-10a6f3f2e214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:58:37.190744 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:37.190574    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04\": container with ID starting with 4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04 not found: ID does not exist" containerID="4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04"
Oct 30 05:58:37.190744 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.190617    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04"} err="failed to get container status \"4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04\": rpc error: code = NotFound desc = could not find container \"4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04\": container with ID starting with 4d43284f6e0b4f164a5942d764dee460cf72c697ca7043203a4a83e5e8b49a04 not found: ID does not exist"
Oct 30 05:58:37.215520 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.215475    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.215743 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.215727    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.215836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.215826    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.215917 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.215906    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.215991 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.215982    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.364600 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.364563    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.364852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.364836    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.364949 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.364938    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.365040 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.365028    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4djs\" (UniqueName: \"kubernetes.io/projected/12fa15c4-7aa4-4664-8a37-10a6f3f2e214-kube-api-access-f4djs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:37.428206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.428104    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35668233-ef08-4087-a61d-f788c405e7d9" path="/var/lib/kubelet/pods/35668233-ef08-4087-a61d-f788c405e7d9/volumes"
Oct 30 05:58:37.429493 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.429000    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:37.430576 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.430557    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80126769-23a0-4d10-8955-bd34746b26cc" path="/var/lib/kubelet/pods/80126769-23a0-4d10-8955-bd34746b26cc/volumes"
Oct 30 05:58:37.559867 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.559825    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"]
Oct 30 05:58:37.575439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.575386    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-dgm95"]
Oct 30 05:58:37.995343 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:37.995307    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:39.111322 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.111294    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:58:39.154642 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.154618    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:58:39.171986 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.171955    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:58:39.189212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.187423    5250 generic.go:356] "Generic (PLEG): container finished" podID="a8de185f-7e92-432a-b994-6673f21895de" containerID="d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e" exitCode=0
Oct 30 05:58:39.189212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.187528    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"
Oct 30 05:58:39.189212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.187543    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" event={"ID":"a8de185f-7e92-432a-b994-6673f21895de","Type":"ContainerDied","Data":"d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e"}
Oct 30 05:58:39.189212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.187618    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb" event={"ID":"a8de185f-7e92-432a-b994-6673f21895de","Type":"ContainerDied","Data":"295108b0333be1caa0ef9a3f1758396de77d96b072c2adec905b06951b1a3aed"}
Oct 30 05:58:39.189212 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.187651    5250 scope.go:122] "RemoveContainer" containerID="d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e"
Oct 30 05:58:39.193307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.190119    5250 generic.go:356] "Generic (PLEG): container finished" podID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerID="ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76" exitCode=0
Oct 30 05:58:39.193307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.190221    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" event={"ID":"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba","Type":"ContainerDied","Data":"ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76"}
Oct 30 05:58:39.193307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.190268    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc" event={"ID":"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba","Type":"ContainerDied","Data":"3d14d48eb4aff2778e5e6aac8eeec9e618c8d6d4c719a0865692633a946b8d98"}
Oct 30 05:58:39.193307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.190353    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"
Oct 30 05:58:39.193307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.192370    5250 generic.go:356] "Generic (PLEG): container finished" podID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerID="41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc" exitCode=0
Oct 30 05:58:39.193307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.192455    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" event={"ID":"ae476bab-f205-47f8-80e2-8d04be4e88e1","Type":"ContainerDied","Data":"41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc"}
Oct 30 05:58:39.193307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.192493    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t" event={"ID":"ae476bab-f205-47f8-80e2-8d04be4e88e1","Type":"ContainerDied","Data":"319c3414b55be7fdccb887df0c7170dd1bccc84a72cfc84a002670ff01a524d6"}
Oct 30 05:58:39.193307 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.192561    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"
Oct 30 05:58:39.202696 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.202667    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-socket\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.202849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.202713    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-envoy\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.202849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.202772    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-podinfo\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.202849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.202799    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-data\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.202849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.202836    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-socket\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.203088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.202867    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-envoy\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.203088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.202916    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-credential-socket\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.203088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.202946    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-certs\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.203088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203055    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae476bab-f205-47f8-80e2-8d04be4e88e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae476bab-f205-47f8-80e2-8d04be4e88e1-istiod-ca-cert\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.203324 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203150    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-podinfo\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.203324 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203252    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-token\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-token\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.203324 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203273    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-socket" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.203324 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203290    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-kube-api-access-n77qf\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-kube-api-access-n77qf\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.203542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203337    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-data\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.203542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203414    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-credential-socket\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.203542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203509    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-credential-socket" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.203542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203521    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-kube-api-access-qlfxs\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-kube-api-access-qlfxs\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.203542 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203521    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-socket" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.203821 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203556    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-certs\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.203821 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203606    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-token\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-token\") pod \"ae476bab-f205-47f8-80e2-8d04be4e88e1\" (UID: \"ae476bab-f205-47f8-80e2-8d04be4e88e1\") "
Oct 30 05:58:39.203821 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203705    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istiod-ca-cert\") pod \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\" (UID: \"e404018b-e1d0-47ee-afcb-0a1cd2cc63ba\") "
Oct 30 05:58:39.203821 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.203720    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-data" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.204193 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.204153    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-certs" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.204565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.204541    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.204641 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.204571    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.204641 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.204586    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.204641 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.204600    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.204641 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.204613    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.204863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.204738    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae476bab-f205-47f8-80e2-8d04be4e88e1-istiod-ca-cert" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:39.204863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.204836    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-certs" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.207298 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.207272    5250 scope.go:122] "RemoveContainer" containerID="d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e"
Oct 30 05:58:39.207452 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.207435    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-kube-api-access-n77qf" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "kube-api-access-n77qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:39.207818 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.207799    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-data" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.207920 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:39.207900    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e\": container with ID starting with d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e not found: ID does not exist" containerID="d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e"
Oct 30 05:58:39.207992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.207933    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e"} err="failed to get container status \"d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e\": rpc error: code = NotFound desc = could not find container \"d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e\": container with ID starting with d26a1e318b9f34492d6c3622599a3a67e5f1e16a963e29a1f540ac7cb819b49e not found: ID does not exist"
Oct 30 05:58:39.207992 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.207957    5250 scope.go:122] "RemoveContainer" containerID="ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76"
Oct 30 05:58:39.208348 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.208330    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istiod-ca-cert" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:39.208942 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.208921    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-credential-socket" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.210385 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.210359    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-token" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:39.210509 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.210490    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-envoy" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.210595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.210518    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-podinfo" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:58:39.210660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.210620    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-podinfo" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:58:39.211037 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.211012    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-envoy" pod "ae476bab-f205-47f8-80e2-8d04be4e88e1" (UID: "ae476bab-f205-47f8-80e2-8d04be4e88e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.212635 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.212611    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-token" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:39.212792 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.212774    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-kube-api-access-qlfxs" pod "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" (UID: "e404018b-e1d0-47ee-afcb-0a1cd2cc63ba"). InnerVolumeSpecName "kube-api-access-qlfxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:39.228386 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.228349    5250 scope.go:122] "RemoveContainer" containerID="ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76"
Oct 30 05:58:39.228888 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:39.228860    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76\": container with ID starting with ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76 not found: ID does not exist" containerID="ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76"
Oct 30 05:58:39.229608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.228901    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76"} err="failed to get container status \"ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76\": rpc error: code = NotFound desc = could not find container \"ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76\": container with ID starting with ea6744ac48685803e6f321485265c75fa03614078ad09cb93a5d46e268354d76 not found: ID does not exist"
Oct 30 05:58:39.229608 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.228932    5250 scope.go:122] "RemoveContainer" containerID="41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc"
Oct 30 05:58:39.261387 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.261352    5250 scope.go:122] "RemoveContainer" containerID="41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc"
Oct 30 05:58:39.261829 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:58:39.261803    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc\": container with ID starting with 41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc not found: ID does not exist" containerID="41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc"
Oct 30 05:58:39.261930 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.261833    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc"} err="failed to get container status \"41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc\": rpc error: code = NotFound desc = could not find container \"41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc\": container with ID starting with 41055b045d5df205a720c05571be051e3b253b1d71a09c3839381ef00c8fd3bc not found: ID does not exist"
Oct 30 05:58:39.298258 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.298217    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" path="/var/lib/kubelet/pods/12fa15c4-7aa4-4664-8a37-10a6f3f2e214/volumes"
Oct 30 05:58:39.305403 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.305380    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-credential-socket\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.305619 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.305599    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-data\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.305778 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.305762    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-envoy\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.305939 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.305923    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8de185f-7e92-432a-b994-6673f21895de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8de185f-7e92-432a-b994-6673f21895de-istiod-ca-cert\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.306135 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.305803    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-data" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.306275 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.306256    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-credential-socket" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.306264    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-certs\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.306739    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-istio-token\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-istio-token\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.306827    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-socket\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.306861    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-kube-api-access-m65ds\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-kube-api-access-m65ds\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.306954    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8de185f-7e92-432a-b994-6673f21895de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8de185f-7e92-432a-b994-6673f21895de-istio-podinfo\") pod \"a8de185f-7e92-432a-b994-6673f21895de\" (UID: \"a8de185f-7e92-432a-b994-6673f21895de\") "
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307694    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307715    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307729    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307743    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae476bab-f205-47f8-80e2-8d04be4e88e1-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307756    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307770    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307786    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n77qf\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-kube-api-access-n77qf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307798    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307811    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae476bab-f205-47f8-80e2-8d04be4e88e1-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307824    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlfxs\" (UniqueName: \"kubernetes.io/projected/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-kube-api-access-qlfxs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307837    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307850    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae476bab-f205-47f8-80e2-8d04be4e88e1-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307862    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307875    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.308269 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.307887    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.309267 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.306558    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-certs" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.309388 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.306617    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8de185f-7e92-432a-b994-6673f21895de-istiod-ca-cert" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:39.309388 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.309119    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-envoy" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.309510 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.309407    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-socket" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:39.310081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.310053    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-istio-token" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:39.311944 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.311915    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-kube-api-access-m65ds" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "kube-api-access-m65ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:39.312468 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.312447    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8de185f-7e92-432a-b994-6673f21895de-istio-podinfo" pod "a8de185f-7e92-432a-b994-6673f21895de" (UID: "a8de185f-7e92-432a-b994-6673f21895de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:58:39.410088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.409995    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.410088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.410033    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8de185f-7e92-432a-b994-6673f21895de-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.410088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.410050    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.410088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.410064    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.410088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.410077    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8de185f-7e92-432a-b994-6673f21895de-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.410088 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.410091    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65ds\" (UniqueName: \"kubernetes.io/projected/a8de185f-7e92-432a-b994-6673f21895de-kube-api-access-m65ds\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.410583 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.410107    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8de185f-7e92-432a-b994-6673f21895de-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:39.541152 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.541115    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"]
Oct 30 05:58:39.545130 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.545099    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pr7tc"]
Oct 30 05:58:39.582099 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.582018    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"]
Oct 30 05:58:39.591362 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.591332    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-w6n5t"]
Oct 30 05:58:39.658357 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.658324    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"]
Oct 30 05:58:39.669565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:39.669534    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-247gb"]
Oct 30 05:58:40.229202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:40.224504    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:40.725955 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:40.725919    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:41.149946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:41.149918    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"]
Oct 30 05:58:41.284610 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:41.284550    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8de185f-7e92-432a-b994-6673f21895de" path="/var/lib/kubelet/pods/a8de185f-7e92-432a-b994-6673f21895de/volumes"
Oct 30 05:58:41.285270 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:41.285244    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" path="/var/lib/kubelet/pods/ae476bab-f205-47f8-80e2-8d04be4e88e1/volumes"
Oct 30 05:58:41.285872 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:41.285854    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" path="/var/lib/kubelet/pods/e404018b-e1d0-47ee-afcb-0a1cd2cc63ba/volumes"
Oct 30 05:58:41.379476 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:41.379447    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:41.578535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:41.578446    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:41.864761 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:41.864579    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"]
Oct 30 05:58:41.953964 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:41.953931    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"]
Oct 30 05:58:42.060309 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:42.052860    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:42.271958 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:42.271926    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:43.064122 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:43.064077    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:44.129338 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:44.129299    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:44.298515 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:44.298472    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:44.462836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:44.449353    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:44.859824 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:44.859738    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:45.007676 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:45.007644    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:45.250001 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:45.249450    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:45.456207 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:45.453807    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:46.865329 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:46.865284    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv" podUID="b3352c23-ff38-4ef7-8995-4db145aed854" containerName="payload-processing" containerID="cri-o://2b1cb8d3e31b19e77ca8e55d414db5ac18f8b3538910e209de8615bf199f09ca" gracePeriod=25
Oct 30 05:58:46.955081 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:46.954996    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk" podUID="fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848" containerName="payload-pre-processing" containerID="cri-o://3faf5a0a8304c2096b088628d0e5bf4cc3b4b94218e5ed61aa8ee44dfb608a4b" gracePeriod=25
Oct 30 05:58:47.244106 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.244073    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:47.308369 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.308336    5250 generic.go:356] "Generic (PLEG): container finished" podID="fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848" containerID="3faf5a0a8304c2096b088628d0e5bf4cc3b4b94218e5ed61aa8ee44dfb608a4b" exitCode=0
Oct 30 05:58:47.308557 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.308522    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk" event={"ID":"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848","Type":"ContainerDied","Data":"3faf5a0a8304c2096b088628d0e5bf4cc3b4b94218e5ed61aa8ee44dfb608a4b"}
Oct 30 05:58:47.310797 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.310768    5250 generic.go:356] "Generic (PLEG): container finished" podID="b3352c23-ff38-4ef7-8995-4db145aed854" containerID="2b1cb8d3e31b19e77ca8e55d414db5ac18f8b3538910e209de8615bf199f09ca" exitCode=0
Oct 30 05:58:47.310952 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.310934    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv" event={"ID":"b3352c23-ff38-4ef7-8995-4db145aed854","Type":"ContainerDied","Data":"2b1cb8d3e31b19e77ca8e55d414db5ac18f8b3538910e209de8615bf199f09ca"}
Oct 30 05:58:47.340734 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.340699    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:58:47.389105 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.389072    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:58:47.446509 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.446410    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-plugins-config-volume\") pod \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\" (UID: \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\") "
Oct 30 05:58:47.446509 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.446504    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3352c23-ff38-4ef7-8995-4db145aed854-kube-api-access-b795w\" (UniqueName: \"kubernetes.io/projected/b3352c23-ff38-4ef7-8995-4db145aed854-kube-api-access-b795w\") pod \"b3352c23-ff38-4ef7-8995-4db145aed854\" (UID: \"b3352c23-ff38-4ef7-8995-4db145aed854\") "
Oct 30 05:58:47.446750 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.446559    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3352c23-ff38-4ef7-8995-4db145aed854-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3352c23-ff38-4ef7-8995-4db145aed854-plugins-config-volume\") pod \"b3352c23-ff38-4ef7-8995-4db145aed854\" (UID: \"b3352c23-ff38-4ef7-8995-4db145aed854\") "
Oct 30 05:58:47.446821 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.446800    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-kube-api-access-kwwzt\" (UniqueName: \"kubernetes.io/projected/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-kube-api-access-kwwzt\") pod \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\" (UID: \"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848\") "
Oct 30 05:58:47.446977 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.446948    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-plugins-config-volume" pod "fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848" (UID: "fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:47.447455 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.447416    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3352c23-ff38-4ef7-8995-4db145aed854-plugins-config-volume" pod "b3352c23-ff38-4ef7-8995-4db145aed854" (UID: "b3352c23-ff38-4ef7-8995-4db145aed854"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:58:47.447911 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.447875    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:47.448017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.447912    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3352c23-ff38-4ef7-8995-4db145aed854-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:47.449931 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.449893    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-kube-api-access-kwwzt" pod "fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848" (UID: "fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848"). InnerVolumeSpecName "kube-api-access-kwwzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:47.450074 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.450050    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3352c23-ff38-4ef7-8995-4db145aed854-kube-api-access-b795w" pod "b3352c23-ff38-4ef7-8995-4db145aed854" (UID: "b3352c23-ff38-4ef7-8995-4db145aed854"). InnerVolumeSpecName "kube-api-access-b795w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:47.549928 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.549895    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwwzt\" (UniqueName: \"kubernetes.io/projected/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848-kube-api-access-kwwzt\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:47.549928 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:47.549924    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b795w\" (UniqueName: \"kubernetes.io/projected/b3352c23-ff38-4ef7-8995-4db145aed854-kube-api-access-b795w\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:48.319829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.319801    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"
Oct 30 05:58:48.319829 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.319822    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk" event={"ID":"fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848","Type":"ContainerDied","Data":"622497c692bcdcc22f5e9dfccdfebac98f6c2575440e4ae09799bb8dfe338894"}
Oct 30 05:58:48.320489 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.319883    5250 scope.go:122] "RemoveContainer" containerID="3faf5a0a8304c2096b088628d0e5bf4cc3b4b94218e5ed61aa8ee44dfb608a4b"
Oct 30 05:58:48.321713 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.321691    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"
Oct 30 05:58:48.321841 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.321688    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv" event={"ID":"b3352c23-ff38-4ef7-8995-4db145aed854","Type":"ContainerDied","Data":"279c771af9f00c89c0a10653b082b0d52ca90a09ad8ea4889c87f80c9ac23db5"}
Oct 30 05:58:48.334335 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.334314    5250 scope.go:122] "RemoveContainer" containerID="2b1cb8d3e31b19e77ca8e55d414db5ac18f8b3538910e209de8615bf199f09ca"
Oct 30 05:58:48.371406 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.371377    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"]
Oct 30 05:58:48.383167 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.383142    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4323f6-d6d4b675b-wsrdv"]
Oct 30 05:58:48.409752 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.409714    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"]
Oct 30 05:58:48.416547 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.416519    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4323f6-b59784cc5-vmmjk"]
Oct 30 05:58:48.997590 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:48.997551    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:49.283293 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:49.283216    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3352c23-ff38-4ef7-8995-4db145aed854" path="/var/lib/kubelet/pods/b3352c23-ff38-4ef7-8995-4db145aed854/volumes"
Oct 30 05:58:49.283657 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:49.283643    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848" path="/var/lib/kubelet/pods/fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848/volumes"
Oct 30 05:58:49.360679 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:49.360629    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:49.511125 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:49.511092    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:50.891929 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:50.891895    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:51.052030 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:51.051998    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:52.472749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:52.472706    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:54.461464 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:54.461429    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:55.999815 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:55.999781    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:56.172527 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.172487    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerName="main" containerID="cri-o://98ade034837cfdd3ac80dc43cf8d333c2e32b7bdb1aeb1578f1633f0fac1d461" gracePeriod=45
Oct 30 05:58:56.440551 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.440503    5250 generic.go:356] "Generic (PLEG): container finished" podID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerID="98ade034837cfdd3ac80dc43cf8d333c2e32b7bdb1aeb1578f1633f0fac1d461" exitCode=0
Oct 30 05:58:56.440714 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.440585    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" event={"ID":"1a6dbe69-607b-4640-bf3a-d5139a746e1c","Type":"ContainerDied","Data":"98ade034837cfdd3ac80dc43cf8d333c2e32b7bdb1aeb1578f1633f0fac1d461"}
Oct 30 05:58:56.489884 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.489852    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:58:56.536384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.536350    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tls-certs\") pod \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") "
Oct 30 05:58:56.536568 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.536445    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-home\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-home\") pod \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") "
Oct 30 05:58:56.536568 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.536521    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-model-cache\") pod \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") "
Oct 30 05:58:56.536568 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.536556    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tmp-dir\") pod \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") "
Oct 30 05:58:56.536753 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.536617    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6dbe69-607b-4640-bf3a-d5139a746e1c-kube-api-access-krm25\" (UniqueName: \"kubernetes.io/projected/1a6dbe69-607b-4640-bf3a-d5139a746e1c-kube-api-access-krm25\") pod \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") "
Oct 30 05:58:56.536753 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.536725    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-dshm\") pod \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\" (UID: \"1a6dbe69-607b-4640-bf3a-d5139a746e1c\") "
Oct 30 05:58:56.536878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.536744    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-home" pod "1a6dbe69-607b-4640-bf3a-d5139a746e1c" (UID: "1a6dbe69-607b-4640-bf3a-d5139a746e1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:56.536878 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.536861    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tmp-dir" pod "1a6dbe69-607b-4640-bf3a-d5139a746e1c" (UID: "1a6dbe69-607b-4640-bf3a-d5139a746e1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:56.537145 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.537119    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-model-cache" pod "1a6dbe69-607b-4640-bf3a-d5139a746e1c" (UID: "1a6dbe69-607b-4640-bf3a-d5139a746e1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:56.537876 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.537848    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:56.537995 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.537883    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:56.537995 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.537897    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:56.539616 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.539564    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-dshm" pod "1a6dbe69-607b-4640-bf3a-d5139a746e1c" (UID: "1a6dbe69-607b-4640-bf3a-d5139a746e1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:58:56.539783 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.539746    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6dbe69-607b-4640-bf3a-d5139a746e1c-kube-api-access-krm25" pod "1a6dbe69-607b-4640-bf3a-d5139a746e1c" (UID: "1a6dbe69-607b-4640-bf3a-d5139a746e1c"). InnerVolumeSpecName "kube-api-access-krm25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:58:56.540230 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.540166    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tls-certs" pod "1a6dbe69-607b-4640-bf3a-d5139a746e1c" (UID: "1a6dbe69-607b-4640-bf3a-d5139a746e1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:58:56.643729 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.643665    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6dbe69-607b-4640-bf3a-d5139a746e1c-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:56.643981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.643735    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krm25\" (UniqueName: \"kubernetes.io/projected/1a6dbe69-607b-4640-bf3a-d5139a746e1c-kube-api-access-krm25\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:56.643981 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:56.643760    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6dbe69-607b-4640-bf3a-d5139a746e1c-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:58:57.377794 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:57.377754    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 05:58:57.449862 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:57.449835    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"
Oct 30 05:58:57.450038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:57.449834    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8" event={"ID":"1a6dbe69-607b-4640-bf3a-d5139a746e1c","Type":"ContainerDied","Data":"3f15ceaa86b3448b2b57aaf43ef93741c67b5c866b1a2ff95646bb2a37a4307a"}
Oct 30 05:58:57.450038 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:57.449955    5250 scope.go:122] "RemoveContainer" containerID="98ade034837cfdd3ac80dc43cf8d333c2e32b7bdb1aeb1578f1633f0fac1d461"
Oct 30 05:58:57.502786 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:57.502755    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"]
Oct 30 05:58:57.514084 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:57.514055    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a1c17ebc/e2e-lifecycle-model-a1c17ebc-kserve-7b6c467b8f-vrmb8"]
Oct 30 05:58:57.666368 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:57.666337    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:58:59.282093 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:59.282060    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" path="/var/lib/kubelet/pods/1a6dbe69-607b-4640-bf3a-d5139a746e1c/volumes"
Oct 30 05:58:59.484138 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:58:59.484102    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:01.205203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:01.204666    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:02.449298 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:59:02.449255    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:59:02.934970 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:02.934923    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:04.894540 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:04.894492    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:06.556565 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:06.556531    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:08.162516 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:08.162475    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:09.728495 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:09.728457    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:09.876342 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:09.876305    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:11.640751 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:11.640707    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:13.179722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:13.179687    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:14.889193 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:14.889141    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:16.592707 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:16.592672    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:18.498078 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:18.498018    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:18.661581 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:18.661542    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:18.834902 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:18.834820    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:19.787305 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:19.787262    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:20.402096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:20.402058    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:22.014336 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:22.014279    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:23.754929 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:23.754893    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:25.723768 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:25.723727    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:27.363108 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:27.363071    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:27.579488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:27.579449    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:27.730383 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:27.730346    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:27.947047 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:27.945945    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"]
Oct 30 05:59:27.947047 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:27.946246    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy" containerID="cri-o://c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b" gracePeriod=30
Oct 30 05:59:28.000937 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:28.000847    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:29.073954 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:29.073914    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:30.815645 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:30.815610    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:31.257939 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:31.257905    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:32.331238 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:32.331197    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:32.378660 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:59:32.378623    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 05:59:33.101766 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.101700    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:33.235191 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.235104    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:59:33.270574 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.270511    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:33.372586 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.372557    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/389eaf27-0be7-44d2-8e5e-01986e050701-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaf27-0be7-44d2-8e5e-01986e050701-istio-podinfo\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.372597    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389eaf27-0be7-44d2-8e5e-01986e050701-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaf27-0be7-44d2-8e5e-01986e050701-istiod-ca-cert\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.372704    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-istio-token\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-istio-token\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.372783    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-kube-api-access-96n45\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-kube-api-access-96n45\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.372919    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-socket\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.372946    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389eaf27-0be7-44d2-8e5e-01986e050701-istiod-ca-cert" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:59:33.373057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.372953    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-envoy\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.373033    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-certs\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.373092    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-credential-socket\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.373163    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-socket" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:33.373443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.373172    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-data\") pod \"389eaf27-0be7-44d2-8e5e-01986e050701\" (UID: \"389eaf27-0be7-44d2-8e5e-01986e050701\") "
Oct 30 05:59:33.373443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.373350    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-certs" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:33.373443 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.373426    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-credential-socket" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:33.374241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.373817    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-data" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:33.374241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.374039    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.374241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.374063    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaf27-0be7-44d2-8e5e-01986e050701-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.374241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.374078    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.374241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.374107    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.374241 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.374121    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.375315 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.375288    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-envoy" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:33.375561 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.375539    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-kube-api-access-96n45" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "kube-api-access-96n45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:59:33.375840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.375823    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-istio-token" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:59:33.375840 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.375823    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/389eaf27-0be7-44d2-8e5e-01986e050701-istio-podinfo" pod "389eaf27-0be7-44d2-8e5e-01986e050701" (UID: "389eaf27-0be7-44d2-8e5e-01986e050701"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:59:33.475382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.475345    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaf27-0be7-44d2-8e5e-01986e050701-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.475382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.475375    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.475382 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.475385    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96n45\" (UniqueName: \"kubernetes.io/projected/389eaf27-0be7-44d2-8e5e-01986e050701-kube-api-access-96n45\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.475664 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.475395    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaf27-0be7-44d2-8e5e-01986e050701-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:33.793638 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.793608    5250 generic.go:356] "Generic (PLEG): container finished" podID="389eaf27-0be7-44d2-8e5e-01986e050701" containerID="c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b" exitCode=0
Oct 30 05:59:33.793825 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.793657    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"
Oct 30 05:59:33.793825 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.793691    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" event={"ID":"389eaf27-0be7-44d2-8e5e-01986e050701","Type":"ContainerDied","Data":"c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b"}
Oct 30 05:59:33.793825 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.793753    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j" event={"ID":"389eaf27-0be7-44d2-8e5e-01986e050701","Type":"ContainerDied","Data":"84238c9c75c9ecfa7662ba0746cc144217a0dad131aa9639d38cba6eb31df235"}
Oct 30 05:59:33.793825 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.793779    5250 scope.go:122] "RemoveContainer" containerID="c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b"
Oct 30 05:59:33.805271 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.805251    5250 scope.go:122] "RemoveContainer" containerID="649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726"
Oct 30 05:59:33.815503 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.815485    5250 scope.go:122] "RemoveContainer" containerID="c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b"
Oct 30 05:59:33.815747 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:59:33.815731    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b\": container with ID starting with c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b not found: ID does not exist" containerID="c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b"
Oct 30 05:59:33.815785 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.815759    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b"} err="failed to get container status \"c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b\": rpc error: code = NotFound desc = could not find container \"c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b\": container with ID starting with c35381b56108850dc8ad266ece2af56fb3382f2c7f81a96e1684a617ad781d7b not found: ID does not exist"
Oct 30 05:59:33.815785 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.815781    5250 scope.go:122] "RemoveContainer" containerID="649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726"
Oct 30 05:59:33.816003 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:59:33.815988    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726\": container with ID starting with 649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726 not found: ID does not exist" containerID="649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726"
Oct 30 05:59:33.816048 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.816007    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726"} err="failed to get container status \"649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726\": rpc error: code = NotFound desc = could not find container \"649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726\": container with ID starting with 649a0699f688b35ed04d556da096d119cf04abc0db769f696ee34128582d1726 not found: ID does not exist"
Oct 30 05:59:33.858660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.858624    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"]
Oct 30 05:59:33.860978 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.860948    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:33.870373 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:33.870345    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1c17ebc-openshift-default-6f97d76d7c-vkg8j"]
Oct 30 05:59:34.027899 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:34.027870    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:34.397619 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:34.397588    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:34.544491 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:34.544457    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"]
Oct 30 05:59:34.544696 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:34.544671    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy" containerID="cri-o://bd53117abd727563e62cb6e59bea3311b0796cd9390deda33c9fea337987640a" gracePeriod=30
Oct 30 05:59:35.127555 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:35.127518    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:35.282026 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:35.281993    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" path="/var/lib/kubelet/pods/389eaf27-0be7-44d2-8e5e-01986e050701/volumes"
Oct 30 05:59:36.873535 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:36.873495    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:37.747823 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:37.747790    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:38.341920 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:38.341883    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:39.075568 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.075521    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:39.303801 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.303771    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:39.454664 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.454635    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:39.845918 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.845890    5250 generic.go:356] "Generic (PLEG): container finished" podID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerID="bd53117abd727563e62cb6e59bea3311b0796cd9390deda33c9fea337987640a" exitCode=0
Oct 30 05:59:39.846087 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.845957    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" event={"ID":"4c3a5c58-566a-4db2-a801-33bd0c7daa5e","Type":"ContainerDied","Data":"bd53117abd727563e62cb6e59bea3311b0796cd9390deda33c9fea337987640a"}
Oct 30 05:59:39.846087 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.846022    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q" event={"ID":"4c3a5c58-566a-4db2-a801-33bd0c7daa5e","Type":"ContainerDied","Data":"28e2156cf2f089ea5a66c56325b13f2bc439ea568f0cc7f46910e6cd6f220e7a"}
Oct 30 05:59:39.846087 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.846046    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28e2156cf2f089ea5a66c56325b13f2bc439ea568f0cc7f46910e6cd6f220e7a"
Oct 30 05:59:39.862952 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.862927    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:59:39.873844 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.873819    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:39.978880 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.978789    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-podinfo\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.978880 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.978851    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-socket\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.978880 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.978872    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-envoy\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.979210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.978918    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-token\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-token\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.979210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.978958    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-kube-api-access-vc9bh\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-kube-api-access-vc9bh\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.979210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.978985    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-certs\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.979210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.979053    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-data\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.979210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.979116    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-credential-socket\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.979210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.979162    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-socket" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:39.979210 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.979208    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istiod-ca-cert\") pod \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\" (UID: \"4c3a5c58-566a-4db2-a801-33bd0c7daa5e\") "
Oct 30 05:59:39.979603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.979392    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-certs" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:39.979603 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.979494    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-data" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:39.979717 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.979694    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-credential-socket" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:39.980242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.980205    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:39.980242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.980235    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:39.980435 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.980250    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:39.980435 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.980263    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:39.980860 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.980835    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istiod-ca-cert" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:59:39.981900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.981877    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-kube-api-access-vc9bh" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "kube-api-access-vc9bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:59:39.981983 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.981953    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-podinfo" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:59:39.982036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.982014    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-envoy" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:39.982132 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:39.982115    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-token" pod "4c3a5c58-566a-4db2-a801-33bd0c7daa5e" (UID: "4c3a5c58-566a-4db2-a801-33bd0c7daa5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:59:40.081514 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.081470    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:40.081514 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.081501    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:40.081514 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.081515    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:40.081767 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.081527    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9bh\" (UniqueName: \"kubernetes.io/projected/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-kube-api-access-vc9bh\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:40.081767 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.081540    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3a5c58-566a-4db2-a801-33bd0c7daa5e-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:40.445402 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.445371    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:40.702450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.702369    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:40.852983 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.852955    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"
Oct 30 05:59:40.858488 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.858462    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:40.968930 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.967946    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"]
Oct 30 05:59:40.973956 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:40.973901    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-66z6q"]
Oct 30 05:59:41.046946 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:41.046908    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:41.192983 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:41.192946    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:41.284915 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:41.284835    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" path="/var/lib/kubelet/pods/4c3a5c58-566a-4db2-a801-33bd0c7daa5e/volumes"
Oct 30 05:59:41.353537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:41.353504    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:41.965506 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:41.965465    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:42.246836 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:42.246741    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:42.423115 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:42.422348    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:42.577200 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:42.577089    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:42.797584 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:42.797528    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:43.058076 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:43.058042    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:45.889848 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:45.889809    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:46.149712 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:46.149628    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:46.346641 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:46.346606    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:47.146537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:47.146507    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:47.411466 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:47.411386    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:47.880963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:47.880890    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:48.030005 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.029967    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"]
Oct 30 05:59:48.030722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030703    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy"
Oct 30 05:59:48.030722 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030723    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030742    5250 cpu_manager.go:414] "containerMap: removing container" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030747    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030764    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerName="main"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030769    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerName="main"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030777    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030783    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030797    5250 cpu_manager.go:414] "containerMap: removing container" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030803    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030812    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b3352c23-ff38-4ef7-8995-4db145aed854" containerName="payload-processing"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030816    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3352c23-ff38-4ef7-8995-4db145aed854" containerName="payload-processing"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030832    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9c1c44db-9f95-42b5-bac6-9a827a1fd006" containerName="maas-api"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030836    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1c44db-9f95-42b5-bac6-9a827a1fd006" containerName="maas-api"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030842    5250 cpu_manager.go:414] "containerMap: removing container" podUID="80126769-23a0-4d10-8955-bd34746b26cc" containerName="maas-api"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030846    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80126769-23a0-4d10-8955-bd34746b26cc" containerName="maas-api"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030860    5250 cpu_manager.go:414] "containerMap: removing container" podUID="fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848" containerName="payload-pre-processing"
Oct 30 05:59:48.030857 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030865    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848" containerName="payload-pre-processing"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030873    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030877    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030884    5250 cpu_manager.go:414] "containerMap: removing container" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030889    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030895    5250 cpu_manager.go:414] "containerMap: removing container" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030900    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030907    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5d769840-311e-429d-9aba-4533ee9f66b7" containerName="maas-api"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030911    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d769840-311e-429d-9aba-4533ee9f66b7" containerName="maas-api"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.030995    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6dbe69-607b-4640-bf3a-d5139a746e1c" containerName="main"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031005    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae476bab-f205-47f8-80e2-8d04be4e88e1" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031014    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8de185f-7e92-432a-b994-6673f21895de" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031019    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3352c23-ff38-4ef7-8995-4db145aed854" containerName="payload-processing"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031026    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031032    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1c44db-9f95-42b5-bac6-9a827a1fd006" containerName="maas-api"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031038    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="80126769-23a0-4d10-8955-bd34746b26cc" containerName="maas-api"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031046    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7cdeb2-808b-4b56-81ad-dc3b0c8bf848" containerName="payload-pre-processing"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031053    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3a5c58-566a-4db2-a801-33bd0c7daa5e" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031058    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="12fa15c4-7aa4-4664-8a37-10a6f3f2e214" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031064    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="35668233-ef08-4087-a61d-f788c405e7d9" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031070    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d769840-311e-429d-9aba-4533ee9f66b7" containerName="maas-api"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031075    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e404018b-e1d0-47ee-afcb-0a1cd2cc63ba" containerName="istio-proxy"
Oct 30 05:59:48.031376 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.031081    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy"
Oct 30 05:59:48.034626 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.034611    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:48.059519 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.059491    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"]
Oct 30 05:59:48.060841 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.060800    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:48.082770 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.082739    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-1b4b7a\""
Oct 30 05:59:48.082913 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.082869    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-1b4b7a-dockercfg-qqh5l\""
Oct 30 05:59:48.187208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.186215    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86txb\" (UniqueName: \"kubernetes.io/projected/8f39b589-6350-4092-b876-456fd1259aa5-kube-api-access-86txb\") pod \"payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft\" (UID: \"8f39b589-6350-4092-b876-456fd1259aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:48.187208 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.186399    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f39b589-6350-4092-b876-456fd1259aa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft\" (UID: \"8f39b589-6350-4092-b876-456fd1259aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:48.298209 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.298146    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f39b589-6350-4092-b876-456fd1259aa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft\" (UID: \"8f39b589-6350-4092-b876-456fd1259aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:48.299648 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.299622    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86txb\" (UniqueName: \"kubernetes.io/projected/8f39b589-6350-4092-b876-456fd1259aa5-kube-api-access-86txb\") pod \"payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft\" (UID: \"8f39b589-6350-4092-b876-456fd1259aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:48.300344 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.299318    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f39b589-6350-4092-b876-456fd1259aa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft\" (UID: \"8f39b589-6350-4092-b876-456fd1259aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:48.337247 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.335197    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86txb\" (UniqueName: \"kubernetes.io/projected/8f39b589-6350-4092-b876-456fd1259aa5-kube-api-access-86txb\") pod \"payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft\" (UID: \"8f39b589-6350-4092-b876-456fd1259aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:48.349757 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.349729    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:48.421988 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.417618    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:48.592283 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.589279    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:48.662980 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.662949    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"]
Oct 30 05:59:48.663289 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:59:48.663267    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f39b589_6350_4092_b876_456fd1259aa5.slice/crio-46c0443af453b3ecc360461c5c0f4781caa0a3a2568a5c711096eb1e0efaf2ca WatchSource:0}: Error finding container 46c0443af453b3ecc360461c5c0f4781caa0a3a2568a5c711096eb1e0efaf2ca: Status 404 returned error can't find the container with id 46c0443af453b3ecc360461c5c0f4781caa0a3a2568a5c711096eb1e0efaf2ca
Oct 30 05:59:48.806669 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.806640    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:48.992204 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:48.983219    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:49.003863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:49.000285    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft" event={"ID":"8f39b589-6350-4092-b876-456fd1259aa5","Type":"ContainerStarted","Data":"1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c"}
Oct 30 05:59:49.003863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:49.000377    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft" event={"ID":"8f39b589-6350-4092-b876-456fd1259aa5","Type":"ContainerStarted","Data":"46c0443af453b3ecc360461c5c0f4781caa0a3a2568a5c711096eb1e0efaf2ca"}
Oct 30 05:59:49.003863 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:49.001664    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 05:59:49.049461 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:49.047895    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft" podStartSLOduration=2.047872449 podStartE2EDuration="2.047872449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:49.041006132 +0000 UTC m=+1703.872043349" watchObservedRunningTime="2026-10-30 05:59:49.047872449 +0000 UTC m=+1703.878909679"
Oct 30 05:59:49.693534 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:49.693495    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:50.561158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:50.561121    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:50.848054 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:50.847947    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:51.229197 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.229148    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:51.433048 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.433014    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:51.444749 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.444721    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"]
Oct 30 05:59:51.445660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.445636    5250 cpu_manager.go:414] "containerMap: removing container" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy"
Oct 30 05:59:51.445660 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.445656    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389eaf27-0be7-44d2-8e5e-01986e050701" containerName="istio-proxy"
Oct 30 05:59:51.449438 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.449420    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:51.467408 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.467376    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"]
Oct 30 05:59:51.473864 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.473836    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-482468\""
Oct 30 05:59:51.535963 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.535556    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"]
Oct 30 05:59:51.541198 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.541158    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:51.561211 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.560212    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtg4\" (UniqueName: \"kubernetes.io/projected/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-kube-api-access-hhtg4\") pod \"payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k\" (UID: \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:51.561211 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.560328    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k\" (UID: \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:51.563194 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.562384    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-482468-dockercfg-phw2h\""
Oct 30 05:59:51.576057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.576006    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"]
Oct 30 05:59:51.583372 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.583343    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"]
Oct 30 05:59:51.584340 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.584270    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy" containerID="cri-o://9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f" gracePeriod=30
Oct 30 05:59:51.593762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.593667    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:51.662439 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.662401    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtg4\" (UniqueName: \"kubernetes.io/projected/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-kube-api-access-hhtg4\") pod \"payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k\" (UID: \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:51.662618 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.662507    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k656l\" (UniqueName: \"kubernetes.io/projected/fb360e09-c857-4ffb-ae6f-c685c24f56d1-kube-api-access-k656l\") pod \"payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262\" (UID: \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:51.662618 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.662552    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k\" (UID: \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:51.662808 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.662784    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb360e09-c857-4ffb-ae6f-c685c24f56d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262\" (UID: \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:51.663593 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.663565    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k\" (UID: \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:51.686595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.686554    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtg4\" (UniqueName: \"kubernetes.io/projected/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-kube-api-access-hhtg4\") pod \"payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k\" (UID: \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:51.742384 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.739641    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:51.781778 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.780520    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:51.794595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.784952    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k656l\" (UniqueName: \"kubernetes.io/projected/fb360e09-c857-4ffb-ae6f-c685c24f56d1-kube-api-access-k656l\") pod \"payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262\" (UID: \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:51.794595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.785092    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb360e09-c857-4ffb-ae6f-c685c24f56d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262\" (UID: \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:51.794595 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.789761    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb360e09-c857-4ffb-ae6f-c685c24f56d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262\" (UID: \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:51.841202 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.836953    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k656l\" (UniqueName: \"kubernetes.io/projected/fb360e09-c857-4ffb-ae6f-c685c24f56d1-kube-api-access-k656l\") pod \"payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262\" (UID: \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:51.875169 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:51.874908    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:52.245400 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:52.245375    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"]
Oct 30 05:59:52.247154 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:59:52.247123    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea91ab0_130d_4bc0_93a9_41a79bfebbe5.slice/crio-3909b501934372e25408d443e60b00122ecae4909556434ade0af9df162c390e WatchSource:0}: Error finding container 3909b501934372e25408d443e60b00122ecae4909556434ade0af9df162c390e: Status 404 returned error can't find the container with id 3909b501934372e25408d443e60b00122ecae4909556434ade0af9df162c390e
Oct 30 05:59:52.282823 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:52.282789    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:52.314602 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:59:52.294164    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb360e09_c857_4ffb_ae6f_c685c24f56d1.slice/crio-3668fa09470e31fc86146608752ff61860e671980e4905878e3fe5441a42d930 WatchSource:0}: Error finding container 3668fa09470e31fc86146608752ff61860e671980e4905878e3fe5441a42d930: Status 404 returned error can't find the container with id 3668fa09470e31fc86146608752ff61860e671980e4905878e3fe5441a42d930
Oct 30 05:59:52.314602 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:52.300202    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"]
Oct 30 05:59:52.459594 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:52.459557    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:53.003900 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.003867    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:53.045849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.045811    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262" event={"ID":"fb360e09-c857-4ffb-ae6f-c685c24f56d1","Type":"ContainerStarted","Data":"e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf"}
Oct 30 05:59:53.046158 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.046134    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262" event={"ID":"fb360e09-c857-4ffb-ae6f-c685c24f56d1","Type":"ContainerStarted","Data":"3668fa09470e31fc86146608752ff61860e671980e4905878e3fe5441a42d930"}
Oct 30 05:59:53.046335 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.046173    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 05:59:53.048675 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.048639    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k" event={"ID":"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5","Type":"ContainerStarted","Data":"4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136"}
Oct 30 05:59:53.048779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.048692    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k" event={"ID":"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5","Type":"ContainerStarted","Data":"3909b501934372e25408d443e60b00122ecae4909556434ade0af9df162c390e"}
Oct 30 05:59:53.048779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.048715    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 05:59:53.088017 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.087957    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262" podStartSLOduration=2.087939801 podStartE2EDuration="2.087939801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:53.086925223 +0000 UTC m=+1707.917962431" watchObservedRunningTime="2026-10-30 05:59:53.087939801 +0000 UTC m=+1707.918977019"
Oct 30 05:59:53.128146 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.128088    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k" podStartSLOduration=2.1280685249999998 podStartE2EDuration="2.128068525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:53.127770205 +0000 UTC m=+1707.958807422" watchObservedRunningTime="2026-10-30 05:59:53.128068525 +0000 UTC m=+1707.959105743"
Oct 30 05:59:53.207571 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.207533    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:53.361791 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.361699    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:53.566381 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:53.566344    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:54.212233 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:54.211560    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:54.405234 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:54.403998    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:54.581015 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:54.580920    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:56.066290 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:56.066251    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:56.265063 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:56.265027    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:56.896746 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:56.896491    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:59:56.990755 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:56.990685    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:57.041203 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041125    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-data\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.041497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041269    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-token\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-token\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.041497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041381    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istiod-ca-cert\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.041497 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041428    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-envoy\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.041779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041545    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-credential-socket\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.041779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041679    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-socket\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.041779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041722    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-kube-api-access-69gb9\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-kube-api-access-69gb9\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.041779 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041767    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-certs\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.042060 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.041798    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-podinfo\") pod \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\" (UID: \"46d3b064-1ad9-48ab-b1f4-9b975b1a321a\") "
Oct 30 05:59:57.043366 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.043336    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istiod-ca-cert" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:59:57.043735 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.043711    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-data" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:57.043895 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.043878    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-socket" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:57.044052 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.044035    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-credential-socket" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:57.044277 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.044247    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-certs" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:57.048056 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.047858    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-podinfo" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:59:57.048056 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.047929    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-envoy" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:59:57.048056 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.048026    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-token" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:59:57.048374 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.048348    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-kube-api-access-69gb9" pod "46d3b064-1ad9-48ab-b1f4-9b975b1a321a" (UID: "46d3b064-1ad9-48ab-b1f4-9b975b1a321a"). InnerVolumeSpecName "kube-api-access-69gb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:59:57.097343 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.097304    5250 generic.go:356] "Generic (PLEG): container finished" podID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerID="9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f" exitCode=0
Oct 30 05:59:57.097852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.097361    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" event={"ID":"46d3b064-1ad9-48ab-b1f4-9b975b1a321a","Type":"ContainerDied","Data":"9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f"}
Oct 30 05:59:57.097852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.097422    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx" event={"ID":"46d3b064-1ad9-48ab-b1f4-9b975b1a321a","Type":"ContainerDied","Data":"4699f561e81946814f728b0a061a4ce9439e5405b414d9b24b5dbe9c4fab5f0a"}
Oct 30 05:59:57.097852 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.097444    5250 scope.go:122] "RemoveContainer" containerID="9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f"
Oct 30 05:59:57.098020 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.097857    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"
Oct 30 05:59:57.124363 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.124340    5250 scope.go:122] "RemoveContainer" containerID="2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989"
Oct 30 05:59:57.136554 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.136526    5250 scope.go:122] "RemoveContainer" containerID="9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f"
Oct 30 05:59:57.136941 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:59:57.136916    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f\": container with ID starting with 9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f not found: ID does not exist" containerID="9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f"
Oct 30 05:59:57.137057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.136952    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f"} err="failed to get container status \"9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f\": rpc error: code = NotFound desc = could not find container \"9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f\": container with ID starting with 9e9f0392edde9b22ca23019ffc0347837a17281143b9c4aeca5dbd799513be4f not found: ID does not exist"
Oct 30 05:59:57.137057 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.136976    5250 scope.go:122] "RemoveContainer" containerID="2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989"
Oct 30 05:59:57.137308 ip-10-0-130-194 kubenswrapper[5250]: E1030 05:59:57.137289    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989\": container with ID starting with 2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989 not found: ID does not exist" containerID="2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989"
Oct 30 05:59:57.137363 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.137319    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989"} err="failed to get container status \"2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989\": rpc error: code = NotFound desc = could not find container \"2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989\": container with ID starting with 2eb814c4d41a9247106e2269daa2f3d70a77053c2a5befedba66259637ba1989 not found: ID does not exist"
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144298    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144331    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144344    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144357    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144370    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144384    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144399    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69gb9\" (UniqueName: \"kubernetes.io/projected/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-kube-api-access-69gb9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144412    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.150244 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.144424    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d3b064-1ad9-48ab-b1f4-9b975b1a321a-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 05:59:57.162531 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.162492    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"]
Oct 30 05:59:57.173974 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.173944    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-76b905-openshift-default-686f48485c-p7ttx"]
Oct 30 05:59:57.284109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.284076    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" path="/var/lib/kubelet/pods/46d3b064-1ad9-48ab-b1f4-9b975b1a321a/volumes"
Oct 30 05:59:57.772036 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.771995    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:57.841096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.841064    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"]
Oct 30 05:59:57.841820 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.841798    5250 cpu_manager.go:414] "containerMap: removing container" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy"
Oct 30 05:59:57.841930 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.841822    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy"
Oct 30 05:59:57.842039 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.842025    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy"
Oct 30 05:59:57.842096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.842057    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy"
Oct 30 05:59:57.847235 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.847216    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:57.863131 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.863104    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1dbaf6-dockercfg-v6g5g\""
Oct 30 05:59:57.867444 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.867418    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"]
Oct 30 05:59:57.899109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.892998    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"]
Oct 30 05:59:57.899109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.894044    5250 cpu_manager.go:414] "containerMap: removing container" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy"
Oct 30 05:59:57.899109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.894064    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d3b064-1ad9-48ab-b1f4-9b975b1a321a" containerName="istio-proxy"
Oct 30 05:59:57.899109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.898459    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:57.899109 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.898975    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1dbaf6\""
Oct 30 05:59:57.914762 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.914732    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"]
Oct 30 05:59:57.951160 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.951127    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:57.955046 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.955019    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skjpm\" (UniqueName: \"kubernetes.io/projected/dd834ca2-0b67-496c-9cb0-bab44ac5e884-kube-api-access-skjpm\") pod \"payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc\" (UID: \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:57.955239 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:57.955220    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd834ca2-0b67-496c-9cb0-bab44ac5e884-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc\" (UID: \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:58.066238 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.066121    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skjpm\" (UniqueName: \"kubernetes.io/projected/dd834ca2-0b67-496c-9cb0-bab44ac5e884-kube-api-access-skjpm\") pod \"payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc\" (UID: \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:58.066524 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.066490    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5\" (UID: \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:58.067265 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.066738    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vssg4\" (UniqueName: \"kubernetes.io/projected/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-kube-api-access-vssg4\") pod \"payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5\" (UID: \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:58.067672 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.067651    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd834ca2-0b67-496c-9cb0-bab44ac5e884-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc\" (UID: \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:58.090919 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.090877    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd834ca2-0b67-496c-9cb0-bab44ac5e884-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc\" (UID: \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:58.115651 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.114885    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skjpm\" (UniqueName: \"kubernetes.io/projected/dd834ca2-0b67-496c-9cb0-bab44ac5e884-kube-api-access-skjpm\") pod \"payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc\" (UID: \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:58.172228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.170249    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5\" (UID: \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:58.172228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.170400    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vssg4\" (UniqueName: \"kubernetes.io/projected/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-kube-api-access-vssg4\") pod \"payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5\" (UID: \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:58.172228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.171816    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5\" (UID: \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:58.172228 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.172007    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:58.217116 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.217086    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vssg4\" (UniqueName: \"kubernetes.io/projected/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-kube-api-access-vssg4\") pod \"payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5\" (UID: \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:58.220048 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.220006    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:58.552450 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.552421    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"]
Oct 30 05:59:58.553212 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:59:58.553165    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd834ca2_0b67_496c_9cb0_bab44ac5e884.slice/crio-16be956f8857258e605e58d6dc4d97b0f893149d0358fbb82d95c6231bc789e1 WatchSource:0}: Error finding container 16be956f8857258e605e58d6dc4d97b0f893149d0358fbb82d95c6231bc789e1: Status 404 returned error can't find the container with id 16be956f8857258e605e58d6dc4d97b0f893149d0358fbb82d95c6231bc789e1
Oct 30 05:59:58.589940 ip-10-0-130-194 kubenswrapper[5250]: W1030 05:59:58.589911    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3d61d5_85f2_4ef2_81ff_e2a0b320a2c2.slice/crio-466c210f15d0cce685920aa2939ccfc57c31f951f216024774db03f6a0a6be03 WatchSource:0}: Error finding container 466c210f15d0cce685920aa2939ccfc57c31f951f216024774db03f6a0a6be03: Status 404 returned error can't find the container with id 466c210f15d0cce685920aa2939ccfc57c31f951f216024774db03f6a0a6be03
Oct 30 05:59:58.591383 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:58.591357    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"]
Oct 30 05:59:59.116849 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.116793    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:59.128904 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.128870    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5" event={"ID":"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2","Type":"ContainerStarted","Data":"e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff"}
Oct 30 05:59:59.129096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.128939    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 05:59:59.129096 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.128963    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5" event={"ID":"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2","Type":"ContainerStarted","Data":"466c210f15d0cce685920aa2939ccfc57c31f951f216024774db03f6a0a6be03"}
Oct 30 05:59:59.131429 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.131400    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc" event={"ID":"dd834ca2-0b67-496c-9cb0-bab44ac5e884","Type":"ContainerStarted","Data":"0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4"}
Oct 30 05:59:59.131537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.131453    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc" event={"ID":"dd834ca2-0b67-496c-9cb0-bab44ac5e884","Type":"ContainerStarted","Data":"16be956f8857258e605e58d6dc4d97b0f893149d0358fbb82d95c6231bc789e1"}
Oct 30 05:59:59.131537 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.131483    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 05:59:59.172632 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.172576    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5" podStartSLOduration=2.172563332 podStartE2EDuration="2.172563332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:59.168802244 +0000 UTC m=+1713.999839461" watchObservedRunningTime="2026-10-30 05:59:59.172563332 +0000 UTC m=+1714.003600549"
Oct 30 05:59:59.213206 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.213139    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc" podStartSLOduration=2.213124001 podStartE2EDuration="2.213124001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:59.208385759 +0000 UTC m=+1714.039422975" watchObservedRunningTime="2026-10-30 05:59:59.213124001 +0000 UTC m=+1714.044161218"
Oct 30 05:59:59.265224 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.265162    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:59.486242 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.482077    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 05:59:59.773686 ip-10-0-130-194 kubenswrapper[5250]: I1030 05:59:59.773587    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:00.176012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.175969    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"]
Oct 30 06:00:00.180588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.180568    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"]
Oct 30 06:00:00.180735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.180703    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:00.184574 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.184556    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:00.207893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.207865    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"]
Oct 30 06:00:00.214521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.214472    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.239286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.239255    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"]
Oct 30 06:00:00.242990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.242961    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"]
Oct 30 06:00:00.250284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.250258    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"]
Oct 30 06:00:00.255960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.255908    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"]
Oct 30 06:00:00.256084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.256025    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:00.273685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.273662    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"]
Oct 30 06:00:00.284308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.284256    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:00.284690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.284660    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"]
Oct 30 06:00:00.293954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.293928    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:00:00.294113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.294093    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-086c60-openshift-default-dockercfg-rnptw\""
Oct 30 06:00:00.301300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.301266    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"]
Oct 30 06:00:00.308582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.308555    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"]
Oct 30 06:00:00.308737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.308678    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:00.312608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.312567    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d60ac-6588-46c3-a44a-3f381e576175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6\" (UID: \"0d2d60ac-6588-46c3-a44a-3f381e576175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:00.312711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.312641    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrg2\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-kube-api-access-vjrg2\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.312780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.312763    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5631a-0e49-442c-bcee-c3324d909233-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d\" (UID: \"34b5631a-0e49-442c-bcee-c3324d909233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:00.312890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.312874    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.312926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.312908    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sld8t\" (UniqueName: \"kubernetes.io/projected/34b5631a-0e49-442c-bcee-c3324d909233-kube-api-access-sld8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d\" (UID: \"34b5631a-0e49-442c-bcee-c3324d909233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:00.312961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.312934    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.313011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.312998    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.313106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.313091    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txtp6\" (UniqueName: \"kubernetes.io/projected/0d2d60ac-6588-46c3-a44a-3f381e576175-kube-api-access-txtp6\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6\" (UID: \"0d2d60ac-6588-46c3-a44a-3f381e576175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:00.313143 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.313121    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.313205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.313172    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.313342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.313320    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.313429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.313370    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.313674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.313655    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"]
Oct 30 06:00:00.313791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.313765    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.313791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.313777    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:00.319743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.319722    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:00.324542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.323978    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"]
Oct 30 06:00:00.336258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.336236    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"]
Oct 30 06:00:00.345230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.342238    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"]
Oct 30 06:00:00.350894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.350867    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"]
Oct 30 06:00:00.359213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.359155    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"]
Oct 30 06:00:00.359339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.359282    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:00.369172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.369143    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"]
Oct 30 06:00:00.415456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.415419    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.415456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.415469    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx\" (UID: \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:00.415726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.415509    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d60ac-6588-46c3-a44a-3f381e576175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6\" (UID: \"0d2d60ac-6588-46c3-a44a-3f381e576175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:00.415726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.415538    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50224823-84da-46b1-9602-3cf83a8e7c27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr\" (UID: \"50224823-84da-46b1-9602-3cf83a8e7c27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:00.415726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.415632    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrg2\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-kube-api-access-vjrg2\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.415972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.415946    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97f4\" (UniqueName: \"kubernetes.io/projected/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-kube-api-access-f97f4\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6\" (UID: \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:00.415972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.415954    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.416110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416040    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ttmm\" (UniqueName: \"kubernetes.io/projected/158297dd-1b28-494d-a338-9305cbc7a461-kube-api-access-5ttmm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq\" (UID: \"158297dd-1b28-494d-a338-9305cbc7a461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:00.416172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416110    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5631a-0e49-442c-bcee-c3324d909233-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d\" (UID: \"34b5631a-0e49-442c-bcee-c3324d909233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:00.416259 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416201    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjpj8\" (UniqueName: \"kubernetes.io/projected/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-kube-api-access-sjpj8\") pod \"maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx\" (UID: \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:00.416326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416260    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhtc\" (UniqueName: \"kubernetes.io/projected/50224823-84da-46b1-9602-3cf83a8e7c27-kube-api-access-jzhtc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr\" (UID: \"50224823-84da-46b1-9602-3cf83a8e7c27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:00.416326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416307    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.416441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416336    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sld8t\" (UniqueName: \"kubernetes.io/projected/34b5631a-0e49-442c-bcee-c3324d909233-kube-api-access-sld8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d\" (UID: \"34b5631a-0e49-442c-bcee-c3324d909233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:00.416441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416364    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.416441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416405    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d60ac-6588-46c3-a44a-3f381e576175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6\" (UID: \"0d2d60ac-6588-46c3-a44a-3f381e576175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:00.416441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416428    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6\" (UID: \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:00.416651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416459    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.416651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416491    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158297dd-1b28-494d-a338-9305cbc7a461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq\" (UID: \"158297dd-1b28-494d-a338-9305cbc7a461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:00.416651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416518    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hr6\" (UniqueName: \"kubernetes.io/projected/5df893bc-ced3-4c63-a549-74f58bcd0423-kube-api-access-65hr6\") pod \"maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf\" (UID: \"5df893bc-ced3-4c63-a549-74f58bcd0423\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:00.416651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416604    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df893bc-ced3-4c63-a549-74f58bcd0423-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf\" (UID: \"5df893bc-ced3-4c63-a549-74f58bcd0423\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:00.416651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416637    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtp6\" (UniqueName: \"kubernetes.io/projected/0d2d60ac-6588-46c3-a44a-3f381e576175-kube-api-access-txtp6\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6\" (UID: \"0d2d60ac-6588-46c3-a44a-3f381e576175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:00.416918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416666    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.416918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416664    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.416918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.416804    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5631a-0e49-442c-bcee-c3324d909233-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d\" (UID: \"34b5631a-0e49-442c-bcee-c3324d909233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:00.417277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.417248    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.417511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.417488    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.417683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.417531    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.417683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.417568    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.417994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.417974    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.418376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.418354    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.419699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.419673    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.420350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.420327    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.446906 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.446832    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.446906 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.446836    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrg2\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-kube-api-access-vjrg2\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.447324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.447304    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sld8t\" (UniqueName: \"kubernetes.io/projected/34b5631a-0e49-442c-bcee-c3324d909233-kube-api-access-sld8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d\" (UID: \"34b5631a-0e49-442c-bcee-c3324d909233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:00.447379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.447309    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtp6\" (UniqueName: \"kubernetes.io/projected/0d2d60ac-6588-46c3-a44a-3f381e576175-kube-api-access-txtp6\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6\" (UID: \"0d2d60ac-6588-46c3-a44a-3f381e576175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:00.497593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.497559    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:00.511808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.511785    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:00.519324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.519263    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx\" (UID: \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:00.519455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.519382    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50224823-84da-46b1-9602-3cf83a8e7c27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr\" (UID: \"50224823-84da-46b1-9602-3cf83a8e7c27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:00.519539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.519453    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97f4\" (UniqueName: \"kubernetes.io/projected/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-kube-api-access-f97f4\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6\" (UID: \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:00.519593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.519547    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ttmm\" (UniqueName: \"kubernetes.io/projected/158297dd-1b28-494d-a338-9305cbc7a461-kube-api-access-5ttmm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq\" (UID: \"158297dd-1b28-494d-a338-9305cbc7a461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:00.519692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.519613    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afaa4574-c8d1-4548-a2c7-42924f05c770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk\" (UID: \"afaa4574-c8d1-4548-a2c7-42924f05c770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:00.520016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.519992    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx\" (UID: \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:00.520217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.520153    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbjg\" (UniqueName: \"kubernetes.io/projected/afaa4574-c8d1-4548-a2c7-42924f05c770-kube-api-access-gzbjg\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk\" (UID: \"afaa4574-c8d1-4548-a2c7-42924f05c770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:00.520303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.520233    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpj8\" (UniqueName: \"kubernetes.io/projected/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-kube-api-access-sjpj8\") pod \"maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx\" (UID: \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:00.520303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.520271    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhtc\" (UniqueName: \"kubernetes.io/projected/50224823-84da-46b1-9602-3cf83a8e7c27-kube-api-access-jzhtc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr\" (UID: \"50224823-84da-46b1-9602-3cf83a8e7c27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:00.520415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.520338    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6\" (UID: \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:00.520415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.520363    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158297dd-1b28-494d-a338-9305cbc7a461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq\" (UID: \"158297dd-1b28-494d-a338-9305cbc7a461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:00.520415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.520386    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hr6\" (UniqueName: \"kubernetes.io/projected/5df893bc-ced3-4c63-a549-74f58bcd0423-kube-api-access-65hr6\") pod \"maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf\" (UID: \"5df893bc-ced3-4c63-a549-74f58bcd0423\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:00.520576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.520430    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df893bc-ced3-4c63-a549-74f58bcd0423-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf\" (UID: \"5df893bc-ced3-4c63-a549-74f58bcd0423\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:00.521102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.521080    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50224823-84da-46b1-9602-3cf83a8e7c27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr\" (UID: \"50224823-84da-46b1-9602-3cf83a8e7c27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:00.521349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.521298    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df893bc-ced3-4c63-a549-74f58bcd0423-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf\" (UID: \"5df893bc-ced3-4c63-a549-74f58bcd0423\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:00.521669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.521647    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6\" (UID: \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:00.521775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.521706    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158297dd-1b28-494d-a338-9305cbc7a461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq\" (UID: \"158297dd-1b28-494d-a338-9305cbc7a461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:00.542717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.542679    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97f4\" (UniqueName: \"kubernetes.io/projected/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-kube-api-access-f97f4\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6\" (UID: \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:00.546670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.546567    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ttmm\" (UniqueName: \"kubernetes.io/projected/158297dd-1b28-494d-a338-9305cbc7a461-kube-api-access-5ttmm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq\" (UID: \"158297dd-1b28-494d-a338-9305cbc7a461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:00.546670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.546581    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpj8\" (UniqueName: \"kubernetes.io/projected/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-kube-api-access-sjpj8\") pod \"maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx\" (UID: \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:00.552955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.552922    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:00.553304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.553270    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhtc\" (UniqueName: \"kubernetes.io/projected/50224823-84da-46b1-9602-3cf83a8e7c27-kube-api-access-jzhtc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr\" (UID: \"50224823-84da-46b1-9602-3cf83a8e7c27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:00.553596 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.553570    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hr6\" (UniqueName: \"kubernetes.io/projected/5df893bc-ced3-4c63-a549-74f58bcd0423-kube-api-access-65hr6\") pod \"maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf\" (UID: \"5df893bc-ced3-4c63-a549-74f58bcd0423\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:00.597278 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.597253    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:00.627504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.627467    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:00.633759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.633724    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afaa4574-c8d1-4548-a2c7-42924f05c770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk\" (UID: \"afaa4574-c8d1-4548-a2c7-42924f05c770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:00.633900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.633828    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbjg\" (UniqueName: \"kubernetes.io/projected/afaa4574-c8d1-4548-a2c7-42924f05c770-kube-api-access-gzbjg\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk\" (UID: \"afaa4574-c8d1-4548-a2c7-42924f05c770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:00.635508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.635479    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afaa4574-c8d1-4548-a2c7-42924f05c770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk\" (UID: \"afaa4574-c8d1-4548-a2c7-42924f05c770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:00.641821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.641642    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:00.657098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.657027    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:00.658854 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.658822    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbjg\" (UniqueName: \"kubernetes.io/projected/afaa4574-c8d1-4548-a2c7-42924f05c770-kube-api-access-gzbjg\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk\" (UID: \"afaa4574-c8d1-4548-a2c7-42924f05c770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:00.669924 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.669723    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:00.680787 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.680586    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:00.833739 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:00.833679    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2d60ac_6588_46c3_a44a_3f381e576175.slice/crio-a6b1c121c00208aa31796902f055d77d9242b96522d3dd9175da0de5ccad11d8 WatchSource:0}: Error finding container a6b1c121c00208aa31796902f055d77d9242b96522d3dd9175da0de5ccad11d8: Status 404 returned error can't find the container with id a6b1c121c00208aa31796902f055d77d9242b96522d3dd9175da0de5ccad11d8
Oct 30 06:00:00.834921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:00.834660    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"]
Oct 30 06:00:01.020574 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.020495    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 06:00:01.072102 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:01.072074    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b5631a_0e49_442c_bcee_c3324d909233.slice/crio-9294f214f41e7c2327328ae705dbef97c779d6baee9e2d920f333eecfe15d17c WatchSource:0}: Error finding container 9294f214f41e7c2327328ae705dbef97c779d6baee9e2d920f333eecfe15d17c: Status 404 returned error can't find the container with id 9294f214f41e7c2327328ae705dbef97c779d6baee9e2d920f333eecfe15d17c
Oct 30 06:00:01.079565 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.079432    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"]
Oct 30 06:00:01.123304 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:01.123265    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dd30bf_0a1e_49a6_83e1_273e9bb3c58e.slice/crio-10e5c96551474b05c83eb7d0dfb753536ff673ff6bb2d9bc2ab9e7517b1af8cd WatchSource:0}: Error finding container 10e5c96551474b05c83eb7d0dfb753536ff673ff6bb2d9bc2ab9e7517b1af8cd: Status 404 returned error can't find the container with id 10e5c96551474b05c83eb7d0dfb753536ff673ff6bb2d9bc2ab9e7517b1af8cd
Oct 30 06:00:01.125806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.125143    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"]
Oct 30 06:00:01.125958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.125805    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:01.126694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.126664    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:01.126824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.126743    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:01.126824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.126787    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:01.151235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.151154    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" event={"ID":"0d2d60ac-6588-46c3-a44a-3f381e576175","Type":"ContainerStarted","Data":"a6b1c121c00208aa31796902f055d77d9242b96522d3dd9175da0de5ccad11d8"}
Oct 30 06:00:01.154058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.154028    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"]
Oct 30 06:00:01.154237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.154216    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" event={"ID":"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e","Type":"ContainerStarted","Data":"10e5c96551474b05c83eb7d0dfb753536ff673ff6bb2d9bc2ab9e7517b1af8cd"}
Oct 30 06:00:01.154994 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:01.154964    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5a32d8_d14e_4a3e_8fd1_c02aaa6aaa04.slice/crio-2dffd9fb48c27d06e177a16b1174b31346c1b3a6dd6ac983a4bbd924a5401c59 WatchSource:0}: Error finding container 2dffd9fb48c27d06e177a16b1174b31346c1b3a6dd6ac983a4bbd924a5401c59: Status 404 returned error can't find the container with id 2dffd9fb48c27d06e177a16b1174b31346c1b3a6dd6ac983a4bbd924a5401c59
Oct 30 06:00:01.158614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.158588    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" event={"ID":"34b5631a-0e49-442c-bcee-c3324d909233","Type":"ContainerStarted","Data":"9294f214f41e7c2327328ae705dbef97c779d6baee9e2d920f333eecfe15d17c"}
Oct 30 06:00:01.203624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.203596    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"]
Oct 30 06:00:01.210737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.210711    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:01.226364 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.226337    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f28e2d\""
Oct 30 06:00:01.226546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.226528    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f28e2d-dockercfg-bpctc\""
Oct 30 06:00:01.231078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.231052    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"]
Oct 30 06:00:01.243661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.242443    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"]
Oct 30 06:00:01.251729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.251703    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"]
Oct 30 06:00:01.255232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.255150    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038e903a-f657-46bd-9b2c-04bd4edfd188-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw\" (UID: \"038e903a-f657-46bd-9b2c-04bd4edfd188\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:01.255390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.255366    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgh5r\" (UniqueName: \"kubernetes.io/projected/038e903a-f657-46bd-9b2c-04bd4edfd188-kube-api-access-fgh5r\") pod \"payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw\" (UID: \"038e903a-f657-46bd-9b2c-04bd4edfd188\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:01.257801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.257449    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:01.257801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.257667    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"]
Oct 30 06:00:01.266306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.266255    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"]
Oct 30 06:00:01.300097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.300068    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:01.301379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.301358    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"]
Oct 30 06:00:01.301514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.301404    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"]
Oct 30 06:00:01.301514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.301424    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"]
Oct 30 06:00:01.376272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.371477    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038e903a-f657-46bd-9b2c-04bd4edfd188-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw\" (UID: \"038e903a-f657-46bd-9b2c-04bd4edfd188\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:01.386227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.385640    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgh5r\" (UniqueName: \"kubernetes.io/projected/038e903a-f657-46bd-9b2c-04bd4edfd188-kube-api-access-fgh5r\") pod \"payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw\" (UID: \"038e903a-f657-46bd-9b2c-04bd4edfd188\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:01.389733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.389704    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038e903a-f657-46bd-9b2c-04bd4edfd188-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw\" (UID: \"038e903a-f657-46bd-9b2c-04bd4edfd188\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:01.471223 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:01.467371    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95aa5335_0ce2_4ae1_a7a1_9c2bc3f81c4c.slice/crio-e00ad7e788c8f78c9bcd57f4cba845fa60433057c534446a358e4915132e4c33 WatchSource:0}: Error finding container e00ad7e788c8f78c9bcd57f4cba845fa60433057c534446a358e4915132e4c33: Status 404 returned error can't find the container with id e00ad7e788c8f78c9bcd57f4cba845fa60433057c534446a358e4915132e4c33
Oct 30 06:00:01.499234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.489721    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sld9\" (UniqueName: \"kubernetes.io/projected/3dcc57d0-a465-4101-94af-ebed0541ca37-kube-api-access-2sld9\") pod \"payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w\" (UID: \"3dcc57d0-a465-4101-94af-ebed0541ca37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:01.499234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.489867    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcc57d0-a465-4101-94af-ebed0541ca37-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w\" (UID: \"3dcc57d0-a465-4101-94af-ebed0541ca37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:01.604350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.582737    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgh5r\" (UniqueName: \"kubernetes.io/projected/038e903a-f657-46bd-9b2c-04bd4edfd188-kube-api-access-fgh5r\") pod \"payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw\" (UID: \"038e903a-f657-46bd-9b2c-04bd4edfd188\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:01.604350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.599749    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcc57d0-a465-4101-94af-ebed0541ca37-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w\" (UID: \"3dcc57d0-a465-4101-94af-ebed0541ca37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:01.604350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.600014    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sld9\" (UniqueName: \"kubernetes.io/projected/3dcc57d0-a465-4101-94af-ebed0541ca37-kube-api-access-2sld9\") pod \"payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w\" (UID: \"3dcc57d0-a465-4101-94af-ebed0541ca37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:01.604350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.600857    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcc57d0-a465-4101-94af-ebed0541ca37-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w\" (UID: \"3dcc57d0-a465-4101-94af-ebed0541ca37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:01.642382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.640848    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sld9\" (UniqueName: \"kubernetes.io/projected/3dcc57d0-a465-4101-94af-ebed0541ca37-kube-api-access-2sld9\") pod \"payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w\" (UID: \"3dcc57d0-a465-4101-94af-ebed0541ca37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:01.719239 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.719202    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:01.829629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.829604    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:01.896859 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:01.896816    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:02.175012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.174967    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" event={"ID":"0d2d60ac-6588-46c3-a44a-3f381e576175","Type":"ContainerStarted","Data":"8cf3b0e01274132874bd980e2989b5971372c8ce52dee8d08f58dac6235ada33"}
Oct 30 06:00:02.175636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.175614    5250 scope.go:122] "RemoveContainer" containerID="8cf3b0e01274132874bd980e2989b5971372c8ce52dee8d08f58dac6235ada33"
Oct 30 06:00:02.177275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.177217    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" event={"ID":"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04","Type":"ContainerStarted","Data":"2dffd9fb48c27d06e177a16b1174b31346c1b3a6dd6ac983a4bbd924a5401c59"}
Oct 30 06:00:02.178929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.178902    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" event={"ID":"158297dd-1b28-494d-a338-9305cbc7a461","Type":"ContainerStarted","Data":"59145f88060195a4ae3c151fe07b8a2ed38e1ab9a13bb20cb149843a184d56d8"}
Oct 30 06:00:02.181333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.181308    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" event={"ID":"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e","Type":"ContainerStarted","Data":"f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23"}
Oct 30 06:00:02.182809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.182784    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" event={"ID":"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c","Type":"ContainerStarted","Data":"e00ad7e788c8f78c9bcd57f4cba845fa60433057c534446a358e4915132e4c33"}
Oct 30 06:00:02.184413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.184387    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" event={"ID":"34b5631a-0e49-442c-bcee-c3324d909233","Type":"ContainerStarted","Data":"9b82ab894894fcceff520d5e34577485116562e51b58fcce2c717ae1b36bfefd"}
Oct 30 06:00:02.185770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.185746    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" event={"ID":"50224823-84da-46b1-9602-3cf83a8e7c27","Type":"ContainerStarted","Data":"00d7a005f0506cfec2d90d584e80ff3bf166f0dbea95e22e19f286e6d2549de6"}
Oct 30 06:00:02.187060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.187038    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" event={"ID":"5df893bc-ced3-4c63-a549-74f58bcd0423","Type":"ContainerStarted","Data":"ca21621729d5e222826a259b693fa6af092c62cd69c669407909966be6dae5cb"}
Oct 30 06:00:02.188219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.188200    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" event={"ID":"afaa4574-c8d1-4548-a2c7-42924f05c770","Type":"ContainerStarted","Data":"bd27a1b64d61beea396abc96c06e3c57629171dbdfa6b4bd95ecb00c94bed053"}
Oct 30 06:00:02.256659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.256599    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" podStartSLOduration=2.256581547 podStartE2EDuration="2.256581547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:02.252894854 +0000 UTC m=+1717.083932071" watchObservedRunningTime="2026-10-30 06:00:02.256581547 +0000 UTC m=+1717.087618764"
Oct 30 06:00:02.310621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.310517    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" podStartSLOduration=2.310495197 podStartE2EDuration="2.310495197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:02.304697489 +0000 UTC m=+1717.135734707" watchObservedRunningTime="2026-10-30 06:00:02.310495197 +0000 UTC m=+1717.141532415"
Oct 30 06:00:02.542491 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:02.541943    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:00:02.553900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.553873    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:02.568046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.566317    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 06:00:02.568046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.566375    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 06:00:02.568046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.567521    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"]
Oct 30 06:00:02.599676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:02.599649    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"]
Oct 30 06:00:02.617215 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:02.617167    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcc57d0_a465_4101_94af_ebed0541ca37.slice/crio-46caee475b75a4096d5d74efc1810adb2f7fa6c7b8b7773fcf62c700120206a9 WatchSource:0}: Error finding container 46caee475b75a4096d5d74efc1810adb2f7fa6c7b8b7773fcf62c700120206a9: Status 404 returned error can't find the container with id 46caee475b75a4096d5d74efc1810adb2f7fa6c7b8b7773fcf62c700120206a9
Oct 30 06:00:03.200079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.200050    5250 generic.go:356] "Generic (PLEG): container finished" podID="158297dd-1b28-494d-a338-9305cbc7a461" containerID="5438d44d9b899124ece9114608cbf59a214d12585d0630255da754c03b7a2182" exitCode=60
Oct 30 06:00:03.200283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.200192    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" event={"ID":"158297dd-1b28-494d-a338-9305cbc7a461","Type":"ContainerDied","Data":"5438d44d9b899124ece9114608cbf59a214d12585d0630255da754c03b7a2182"}
Oct 30 06:00:03.200567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.200549    5250 scope.go:122] "RemoveContainer" containerID="5438d44d9b899124ece9114608cbf59a214d12585d0630255da754c03b7a2182"
Oct 30 06:00:03.202310 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.202285    5250 generic.go:356] "Generic (PLEG): container finished" podID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerID="a6f039c15a87f0c99007826201e2dc1440d008280f0697a158b11d0b212641e1" exitCode=60
Oct 30 06:00:03.202435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.202373    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" event={"ID":"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c","Type":"ContainerDied","Data":"a6f039c15a87f0c99007826201e2dc1440d008280f0697a158b11d0b212641e1"}
Oct 30 06:00:03.202699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.202675    5250 scope.go:122] "RemoveContainer" containerID="a6f039c15a87f0c99007826201e2dc1440d008280f0697a158b11d0b212641e1"
Oct 30 06:00:03.204159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.204141    5250 generic.go:356] "Generic (PLEG): container finished" podID="34b5631a-0e49-442c-bcee-c3324d909233" containerID="9b82ab894894fcceff520d5e34577485116562e51b58fcce2c717ae1b36bfefd" exitCode=60
Oct 30 06:00:03.204268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.204223    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" event={"ID":"34b5631a-0e49-442c-bcee-c3324d909233","Type":"ContainerDied","Data":"9b82ab894894fcceff520d5e34577485116562e51b58fcce2c717ae1b36bfefd"}
Oct 30 06:00:03.204540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.204498    5250 scope.go:122] "RemoveContainer" containerID="9b82ab894894fcceff520d5e34577485116562e51b58fcce2c717ae1b36bfefd"
Oct 30 06:00:03.206291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.206262    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" event={"ID":"50224823-84da-46b1-9602-3cf83a8e7c27","Type":"ContainerStarted","Data":"175685b0e3ec22cb1bc3b46fc2db1723da6826e626d94fa53316094a45077b9c"}
Oct 30 06:00:03.207713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.207693    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" event={"ID":"5df893bc-ced3-4c63-a549-74f58bcd0423","Type":"ContainerStarted","Data":"3e50459ecf39300682f049ec3f647ceba21783c5bd94a6995feda78298c4edf6"}
Oct 30 06:00:03.207905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.207887    5250 scope.go:122] "RemoveContainer" containerID="3e50459ecf39300682f049ec3f647ceba21783c5bd94a6995feda78298c4edf6"
Oct 30 06:00:03.209334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.209314    5250 generic.go:356] "Generic (PLEG): container finished" podID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerID="0569b97218c34e8795e84a3884c9307451cbf9a555f83c54cc7f179728d2eede" exitCode=60
Oct 30 06:00:03.209422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.209386    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" event={"ID":"afaa4574-c8d1-4548-a2c7-42924f05c770","Type":"ContainerDied","Data":"0569b97218c34e8795e84a3884c9307451cbf9a555f83c54cc7f179728d2eede"}
Oct 30 06:00:03.209595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.209581    5250 scope.go:122] "RemoveContainer" containerID="0569b97218c34e8795e84a3884c9307451cbf9a555f83c54cc7f179728d2eede"
Oct 30 06:00:03.211263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.211059    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw" event={"ID":"038e903a-f657-46bd-9b2c-04bd4edfd188","Type":"ContainerStarted","Data":"e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14"}
Oct 30 06:00:03.211263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.211102    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw" event={"ID":"038e903a-f657-46bd-9b2c-04bd4edfd188","Type":"ContainerStarted","Data":"dc3c606c0ebdfba3f1e6ee2db766970165c838e6013c717be8bc54983dc913a2"}
Oct 30 06:00:03.211263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.211145    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:03.213114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.213069    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w" event={"ID":"3dcc57d0-a465-4101-94af-ebed0541ca37","Type":"ContainerStarted","Data":"dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8"}
Oct 30 06:00:03.213233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.213115    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w" event={"ID":"3dcc57d0-a465-4101-94af-ebed0541ca37","Type":"ContainerStarted","Data":"46caee475b75a4096d5d74efc1810adb2f7fa6c7b8b7773fcf62c700120206a9"}
Oct 30 06:00:03.214049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.213904    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:03.219373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.218647    5250 generic.go:356] "Generic (PLEG): container finished" podID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerID="8cf3b0e01274132874bd980e2989b5971372c8ce52dee8d08f58dac6235ada33" exitCode=60
Oct 30 06:00:03.219373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.218736    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" event={"ID":"0d2d60ac-6588-46c3-a44a-3f381e576175","Type":"ContainerDied","Data":"8cf3b0e01274132874bd980e2989b5971372c8ce52dee8d08f58dac6235ada33"}
Oct 30 06:00:03.219373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.218775    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" event={"ID":"0d2d60ac-6588-46c3-a44a-3f381e576175","Type":"ContainerStarted","Data":"282d32528083694e2dc9f45cb9c89e14822cd753f13a14b6cac1ec0f70750356"}
Oct 30 06:00:03.222239 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.222212    5250 generic.go:356] "Generic (PLEG): container finished" podID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerID="0e8660e3943eb531d008295ffd6422951cf00949fb959bfd32a00259a00890b0" exitCode=60
Oct 30 06:00:03.222529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.222326    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" event={"ID":"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04","Type":"ContainerDied","Data":"0e8660e3943eb531d008295ffd6422951cf00949fb959bfd32a00259a00890b0"}
Oct 30 06:00:03.223284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.223263    5250 scope.go:122] "RemoveContainer" containerID="0e8660e3943eb531d008295ffd6422951cf00949fb959bfd32a00259a00890b0"
Oct 30 06:00:03.335125 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.335073    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw" podStartSLOduration=2.335060198 podStartE2EDuration="2.335060198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.286653825 +0000 UTC m=+1718.117691043" watchObservedRunningTime="2026-10-30 06:00:03.335060198 +0000 UTC m=+1718.166097445"
Oct 30 06:00:03.432148 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.432097    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" podStartSLOduration=3.432081437 podStartE2EDuration="3.432081437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.429141609 +0000 UTC m=+1718.260178825" watchObservedRunningTime="2026-10-30 06:00:03.432081437 +0000 UTC m=+1718.263118653"
Oct 30 06:00:03.528114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.528066    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" podStartSLOduration=3.5280526 podStartE2EDuration="3.5280526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.523697487 +0000 UTC m=+1718.354734704" watchObservedRunningTime="2026-10-30 06:00:03.5280526 +0000 UTC m=+1718.359089873"
Oct 30 06:00:03.553478 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.553444    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 06:00:03.553646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.553500    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 06:00:03.792367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.783049    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:03.852491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.852436    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w" podStartSLOduration=2.852418778 podStartE2EDuration="2.852418778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.678923172 +0000 UTC m=+1718.509960390" watchObservedRunningTime="2026-10-30 06:00:03.852418778 +0000 UTC m=+1718.683455994"
Oct 30 06:00:03.887923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.887876    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"]
Oct 30 06:00:03.898070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.896557    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"]
Oct 30 06:00:03.898070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.896670    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:03.919982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.919730    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-765de9-dockercfg-mljcr\""
Oct 30 06:00:03.920118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.919977    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-765de9\""
Oct 30 06:00:03.923195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.921458    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"]
Oct 30 06:00:03.944628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.944584    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:03.951006 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:03.950974    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:04.012702 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.012673    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"]
Oct 30 06:00:04.055065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.054972    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5pp9\" (UniqueName: \"kubernetes.io/projected/266ffdf0-d47f-4ffa-a023-3d939555934d-kube-api-access-h5pp9\") pod \"payload-processing-e2e-inf-b-765de9-7675966898-l2jwg\" (UID: \"266ffdf0-d47f-4ffa-a023-3d939555934d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:04.055354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.055332    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ffdf0-d47f-4ffa-a023-3d939555934d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765de9-7675966898-l2jwg\" (UID: \"266ffdf0-d47f-4ffa-a023-3d939555934d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:04.058211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.055605    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c56799-0c32-4d43-97ae-0912b9ae86ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb\" (UID: \"08c56799-0c32-4d43-97ae-0912b9ae86ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:04.058547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.058524    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjnx\" (UniqueName: \"kubernetes.io/projected/08c56799-0c32-4d43-97ae-0912b9ae86ec-kube-api-access-9cjnx\") pod \"payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb\" (UID: \"08c56799-0c32-4d43-97ae-0912b9ae86ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:04.098206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.097459    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 06:00:04.098773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.098749    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 06:00:04.210340 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.200068    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ffdf0-d47f-4ffa-a023-3d939555934d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765de9-7675966898-l2jwg\" (UID: \"266ffdf0-d47f-4ffa-a023-3d939555934d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:04.210939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.210904    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c56799-0c32-4d43-97ae-0912b9ae86ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb\" (UID: \"08c56799-0c32-4d43-97ae-0912b9ae86ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:04.211406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.211382    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjnx\" (UniqueName: \"kubernetes.io/projected/08c56799-0c32-4d43-97ae-0912b9ae86ec-kube-api-access-9cjnx\") pod \"payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb\" (UID: \"08c56799-0c32-4d43-97ae-0912b9ae86ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:04.211712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.211692    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5pp9\" (UniqueName: \"kubernetes.io/projected/266ffdf0-d47f-4ffa-a023-3d939555934d-kube-api-access-h5pp9\") pod \"payload-processing-e2e-inf-b-765de9-7675966898-l2jwg\" (UID: \"266ffdf0-d47f-4ffa-a023-3d939555934d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:04.213088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.213065    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ffdf0-d47f-4ffa-a023-3d939555934d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765de9-7675966898-l2jwg\" (UID: \"266ffdf0-d47f-4ffa-a023-3d939555934d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:04.222686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.222657    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c56799-0c32-4d43-97ae-0912b9ae86ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb\" (UID: \"08c56799-0c32-4d43-97ae-0912b9ae86ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:04.240010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.239880    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5pp9\" (UniqueName: \"kubernetes.io/projected/266ffdf0-d47f-4ffa-a023-3d939555934d-kube-api-access-h5pp9\") pod \"payload-processing-e2e-inf-b-765de9-7675966898-l2jwg\" (UID: \"266ffdf0-d47f-4ffa-a023-3d939555934d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:04.247925 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.247897    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjnx\" (UniqueName: \"kubernetes.io/projected/08c56799-0c32-4d43-97ae-0912b9ae86ec-kube-api-access-9cjnx\") pod \"payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb\" (UID: \"08c56799-0c32-4d43-97ae-0912b9ae86ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:04.250708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.248387    5250 generic.go:356] "Generic (PLEG): container finished" podID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerID="282d32528083694e2dc9f45cb9c89e14822cd753f13a14b6cac1ec0f70750356" exitCode=60
Oct 30 06:00:04.250708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.248500    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" event={"ID":"0d2d60ac-6588-46c3-a44a-3f381e576175","Type":"ContainerDied","Data":"282d32528083694e2dc9f45cb9c89e14822cd753f13a14b6cac1ec0f70750356"}
Oct 30 06:00:04.250708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.248564    5250 scope.go:122] "RemoveContainer" containerID="8cf3b0e01274132874bd980e2989b5971372c8ce52dee8d08f58dac6235ada33"
Oct 30 06:00:04.250708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.249117    5250 scope.go:122] "RemoveContainer" containerID="282d32528083694e2dc9f45cb9c89e14822cd753f13a14b6cac1ec0f70750356"
Oct 30 06:00:04.250708 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:04.249571    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6_odh-ai-gateway-infra(0d2d60ac-6588-46c3-a44a-3f381e576175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175"
Oct 30 06:00:04.254229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.254203    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:04.327227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.327119    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:04.436203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.430594    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:04.436203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.433197    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" event={"ID":"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04","Type":"ContainerStarted","Data":"8e398729d863a82ef6b35af1ea08654fd80cae8e5a2bc356c13014da20a1c903"}
Oct 30 06:00:04.538392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.538041    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" event={"ID":"158297dd-1b28-494d-a338-9305cbc7a461","Type":"ContainerStarted","Data":"42e313e08754dafc6a538783853d475645a360abe386bf1ffc69af2ac36e3808"}
Oct 30 06:00:04.542441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.542241    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" event={"ID":"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c","Type":"ContainerStarted","Data":"ce74a63626523b98eba7a626e6fcccc5684162c3173271a7d30519679357bbe8"}
Oct 30 06:00:04.542727 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.542683    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" podStartSLOduration=4.542665578 podStartE2EDuration="4.542665578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:04.534808747 +0000 UTC m=+1719.365845966" watchObservedRunningTime="2026-10-30 06:00:04.542665578 +0000 UTC m=+1719.373702796"
Oct 30 06:00:04.554762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.554731    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 06:00:04.560743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.560712    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 06:00:04.566840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.557132    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" event={"ID":"34b5631a-0e49-442c-bcee-c3324d909233","Type":"ContainerStarted","Data":"3646c9d40a4a3fada2f26db048c21c32316ce5bcffd8960b0e7f1d650f2637ff"}
Oct 30 06:00:04.567057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.567033    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" event={"ID":"50224823-84da-46b1-9602-3cf83a8e7c27","Type":"ContainerDied","Data":"175685b0e3ec22cb1bc3b46fc2db1723da6826e626d94fa53316094a45077b9c"}
Oct 30 06:00:04.567316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.566651    5250 scope.go:122] "RemoveContainer" containerID="175685b0e3ec22cb1bc3b46fc2db1723da6826e626d94fa53316094a45077b9c"
Oct 30 06:00:04.568639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.566113    5250 generic.go:356] "Generic (PLEG): container finished" podID="50224823-84da-46b1-9602-3cf83a8e7c27" containerID="175685b0e3ec22cb1bc3b46fc2db1723da6826e626d94fa53316094a45077b9c" exitCode=60
Oct 30 06:00:04.572158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.570910    5250 generic.go:356] "Generic (PLEG): container finished" podID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerID="3e50459ecf39300682f049ec3f647ceba21783c5bd94a6995feda78298c4edf6" exitCode=60
Oct 30 06:00:04.572158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.570991    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" event={"ID":"5df893bc-ced3-4c63-a549-74f58bcd0423","Type":"ContainerDied","Data":"3e50459ecf39300682f049ec3f647ceba21783c5bd94a6995feda78298c4edf6"}
Oct 30 06:00:04.572158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.571031    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" event={"ID":"5df893bc-ced3-4c63-a549-74f58bcd0423","Type":"ContainerStarted","Data":"3f13dd03925560145f7ddd5bc10ca11fe68ca7ced403e50a8e06fac5f81b0852"}
Oct 30 06:00:04.630711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.614128    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" podStartSLOduration=4.614108925 podStartE2EDuration="4.614108925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:04.605914836 +0000 UTC m=+1719.436952054" watchObservedRunningTime="2026-10-30 06:00:04.614108925 +0000 UTC m=+1719.445146142"
Oct 30 06:00:04.636520 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.636488    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" event={"ID":"afaa4574-c8d1-4548-a2c7-42924f05c770","Type":"ContainerStarted","Data":"967adc9ee145b7d799a4263ea2681fa5eee744abbf062299d5ca2d6df8348ec8"}
Oct 30 06:00:04.709508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.708824    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" podStartSLOduration=4.708809502 podStartE2EDuration="4.708809502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:04.651617307 +0000 UTC m=+1719.482654525" watchObservedRunningTime="2026-10-30 06:00:04.708809502 +0000 UTC m=+1719.539846718"
Oct 30 06:00:04.831258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.831121    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" podStartSLOduration=4.831104911 podStartE2EDuration="4.831104911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:04.829711622 +0000 UTC m=+1719.660748842" watchObservedRunningTime="2026-10-30 06:00:04.831104911 +0000 UTC m=+1719.662142129"
Oct 30 06:00:04.887431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.887382    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" podStartSLOduration=4.887361501 podStartE2EDuration="4.887361501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:04.88562485 +0000 UTC m=+1719.716662069" watchObservedRunningTime="2026-10-30 06:00:04.887361501 +0000 UTC m=+1719.718398719"
Oct 30 06:00:04.898614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.898586    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"]
Oct 30 06:00:04.900279 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:04.900252    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266ffdf0_d47f_4ffa_a023_3d939555934d.slice/crio-f5a24fcf30a44d9fb5f1223a8baa5e7a8d5e7e030577608e3101063ddd0f6dbc WatchSource:0}: Error finding container f5a24fcf30a44d9fb5f1223a8baa5e7a8d5e7e030577608e3101063ddd0f6dbc: Status 404 returned error can't find the container with id f5a24fcf30a44d9fb5f1223a8baa5e7a8d5e7e030577608e3101063ddd0f6dbc
Oct 30 06:00:04.987215 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:04.987164    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c56799_0c32_4d43_97ae_0912b9ae86ec.slice/crio-e63874d321891d5f057f0595e25150ea5f753f679cc7388f6f4728ad9532caaf WatchSource:0}: Error finding container e63874d321891d5f057f0595e25150ea5f753f679cc7388f6f4728ad9532caaf: Status 404 returned error can't find the container with id e63874d321891d5f057f0595e25150ea5f753f679cc7388f6f4728ad9532caaf
Oct 30 06:00:04.987380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:04.987309    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"]
Oct 30 06:00:05.553825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.553787    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 06:00:05.554248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.553852    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 06:00:05.645593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.645559    5250 generic.go:356] "Generic (PLEG): container finished" podID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerID="ce74a63626523b98eba7a626e6fcccc5684162c3173271a7d30519679357bbe8" exitCode=60
Oct 30 06:00:05.645761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.645637    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" event={"ID":"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c","Type":"ContainerDied","Data":"ce74a63626523b98eba7a626e6fcccc5684162c3173271a7d30519679357bbe8"}
Oct 30 06:00:05.645761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.645707    5250 scope.go:122] "RemoveContainer" containerID="a6f039c15a87f0c99007826201e2dc1440d008280f0697a158b11d0b212641e1"
Oct 30 06:00:05.646035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.646014    5250 scope.go:122] "RemoveContainer" containerID="ce74a63626523b98eba7a626e6fcccc5684162c3173271a7d30519679357bbe8"
Oct 30 06:00:05.646438 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:05.646413    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx_odh-ai-gateway-infra(95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c"
Oct 30 06:00:05.647726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.647705    5250 generic.go:356] "Generic (PLEG): container finished" podID="34b5631a-0e49-442c-bcee-c3324d909233" containerID="3646c9d40a4a3fada2f26db048c21c32316ce5bcffd8960b0e7f1d650f2637ff" exitCode=60
Oct 30 06:00:05.647834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.647770    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" event={"ID":"34b5631a-0e49-442c-bcee-c3324d909233","Type":"ContainerDied","Data":"3646c9d40a4a3fada2f26db048c21c32316ce5bcffd8960b0e7f1d650f2637ff"}
Oct 30 06:00:05.648040 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.648025    5250 scope.go:122] "RemoveContainer" containerID="3646c9d40a4a3fada2f26db048c21c32316ce5bcffd8960b0e7f1d650f2637ff"
Oct 30 06:00:05.648439 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:05.648417    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d_odh-ai-gateway-infra(34b5631a-0e49-442c-bcee-c3324d909233)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" podUID="34b5631a-0e49-442c-bcee-c3324d909233"
Oct 30 06:00:05.649797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.649774    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" event={"ID":"50224823-84da-46b1-9602-3cf83a8e7c27","Type":"ContainerStarted","Data":"047993f617dd83273f7f06e7a3a27de6d35cd5c47b99656636a033e44b82a542"}
Oct 30 06:00:05.651622 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.651600    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg" event={"ID":"266ffdf0-d47f-4ffa-a023-3d939555934d","Type":"ContainerStarted","Data":"3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb"}
Oct 30 06:00:05.651730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.651638    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg" event={"ID":"266ffdf0-d47f-4ffa-a023-3d939555934d","Type":"ContainerStarted","Data":"f5a24fcf30a44d9fb5f1223a8baa5e7a8d5e7e030577608e3101063ddd0f6dbc"}
Oct 30 06:00:05.651730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.651664    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:05.653686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.653653    5250 generic.go:356] "Generic (PLEG): container finished" podID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerID="3f13dd03925560145f7ddd5bc10ca11fe68ca7ced403e50a8e06fac5f81b0852" exitCode=60
Oct 30 06:00:05.653762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.653732    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" event={"ID":"5df893bc-ced3-4c63-a549-74f58bcd0423","Type":"ContainerDied","Data":"3f13dd03925560145f7ddd5bc10ca11fe68ca7ced403e50a8e06fac5f81b0852"}
Oct 30 06:00:05.653954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.653938    5250 scope.go:122] "RemoveContainer" containerID="3f13dd03925560145f7ddd5bc10ca11fe68ca7ced403e50a8e06fac5f81b0852"
Oct 30 06:00:05.654254 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:05.654224    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf_odh-ai-gateway-infra(5df893bc-ced3-4c63-a549-74f58bcd0423)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423"
Oct 30 06:00:05.660202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.656226    5250 generic.go:356] "Generic (PLEG): container finished" podID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerID="967adc9ee145b7d799a4263ea2681fa5eee744abbf062299d5ca2d6df8348ec8" exitCode=60
Oct 30 06:00:05.660202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.656413    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" event={"ID":"afaa4574-c8d1-4548-a2c7-42924f05c770","Type":"ContainerDied","Data":"967adc9ee145b7d799a4263ea2681fa5eee744abbf062299d5ca2d6df8348ec8"}
Oct 30 06:00:05.660202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.656784    5250 scope.go:122] "RemoveContainer" containerID="967adc9ee145b7d799a4263ea2681fa5eee744abbf062299d5ca2d6df8348ec8"
Oct 30 06:00:05.660202 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:05.657084    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk_odh-ai-gateway-infra(afaa4574-c8d1-4548-a2c7-42924f05c770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770"
Oct 30 06:00:05.660730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.660711    5250 scope.go:122] "RemoveContainer" containerID="282d32528083694e2dc9f45cb9c89e14822cd753f13a14b6cac1ec0f70750356"
Oct 30 06:00:05.661037 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:05.661010    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6_odh-ai-gateway-infra(0d2d60ac-6588-46c3-a44a-3f381e576175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175"
Oct 30 06:00:05.662249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.662226    5250 generic.go:356] "Generic (PLEG): container finished" podID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerID="8e398729d863a82ef6b35af1ea08654fd80cae8e5a2bc356c13014da20a1c903" exitCode=60
Oct 30 06:00:05.662433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.662408    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" event={"ID":"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04","Type":"ContainerDied","Data":"8e398729d863a82ef6b35af1ea08654fd80cae8e5a2bc356c13014da20a1c903"}
Oct 30 06:00:05.662837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.662766    5250 scope.go:122] "RemoveContainer" containerID="8e398729d863a82ef6b35af1ea08654fd80cae8e5a2bc356c13014da20a1c903"
Oct 30 06:00:05.663126 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:05.663103    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6_odh-ai-gateway-infra(ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04"
Oct 30 06:00:05.665736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.665708    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb" event={"ID":"08c56799-0c32-4d43-97ae-0912b9ae86ec","Type":"ContainerStarted","Data":"31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa"}
Oct 30 06:00:05.665833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.665758    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb" event={"ID":"08c56799-0c32-4d43-97ae-0912b9ae86ec","Type":"ContainerStarted","Data":"e63874d321891d5f057f0595e25150ea5f753f679cc7388f6f4728ad9532caaf"}
Oct 30 06:00:05.672945 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.666382    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:05.685328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.685296    5250 generic.go:356] "Generic (PLEG): container finished" podID="158297dd-1b28-494d-a338-9305cbc7a461" containerID="42e313e08754dafc6a538783853d475645a360abe386bf1ffc69af2ac36e3808" exitCode=60
Oct 30 06:00:05.685597 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.685577    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" event={"ID":"158297dd-1b28-494d-a338-9305cbc7a461","Type":"ContainerDied","Data":"42e313e08754dafc6a538783853d475645a360abe386bf1ffc69af2ac36e3808"}
Oct 30 06:00:05.688284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.686139    5250 scope.go:122] "RemoveContainer" containerID="42e313e08754dafc6a538783853d475645a360abe386bf1ffc69af2ac36e3808"
Oct 30 06:00:05.688284 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:05.687589    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq_odh-ai-gateway-infra(158297dd-1b28-494d-a338-9305cbc7a461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" podUID="158297dd-1b28-494d-a338-9305cbc7a461"
Oct 30 06:00:05.693394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.693377    5250 scope.go:122] "RemoveContainer" containerID="9b82ab894894fcceff520d5e34577485116562e51b58fcce2c717ae1b36bfefd"
Oct 30 06:00:05.709269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.709250    5250 scope.go:122] "RemoveContainer" containerID="3e50459ecf39300682f049ec3f647ceba21783c5bd94a6995feda78298c4edf6"
Oct 30 06:00:05.724375 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.724319    5250 scope.go:122] "RemoveContainer" containerID="0569b97218c34e8795e84a3884c9307451cbf9a555f83c54cc7f179728d2eede"
Oct 30 06:00:05.738875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.738856    5250 scope.go:122] "RemoveContainer" containerID="0e8660e3943eb531d008295ffd6422951cf00949fb959bfd32a00259a00890b0"
Oct 30 06:00:05.753260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.753160    5250 scope.go:122] "RemoveContainer" containerID="5438d44d9b899124ece9114608cbf59a214d12585d0630255da754c03b7a2182"
Oct 30 06:00:05.926831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:05.926780    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb" podStartSLOduration=2.926766218 podStartE2EDuration="2.926766218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:05.921435213 +0000 UTC m=+1720.752472429" watchObservedRunningTime="2026-10-30 06:00:05.926766218 +0000 UTC m=+1720.757803472"
Oct 30 06:00:06.188223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.185051    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg" podStartSLOduration=3.185030796 podStartE2EDuration="3.185030796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:06.183581102 +0000 UTC m=+1721.014618330" watchObservedRunningTime="2026-10-30 06:00:06.185030796 +0000 UTC m=+1721.016068015"
Oct 30 06:00:06.308851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.308812    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:06.554421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.554319    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 06:00:06.554421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.554375    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 06:00:06.700981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.700953    5250 scope.go:122] "RemoveContainer" containerID="3646c9d40a4a3fada2f26db048c21c32316ce5bcffd8960b0e7f1d650f2637ff"
Oct 30 06:00:06.701430 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:06.701407    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d_odh-ai-gateway-infra(34b5631a-0e49-442c-bcee-c3324d909233)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" podUID="34b5631a-0e49-442c-bcee-c3324d909233"
Oct 30 06:00:06.702614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.702591    5250 generic.go:356] "Generic (PLEG): container finished" podID="50224823-84da-46b1-9602-3cf83a8e7c27" containerID="047993f617dd83273f7f06e7a3a27de6d35cd5c47b99656636a033e44b82a542" exitCode=60
Oct 30 06:00:06.702725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.702668    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" event={"ID":"50224823-84da-46b1-9602-3cf83a8e7c27","Type":"ContainerDied","Data":"047993f617dd83273f7f06e7a3a27de6d35cd5c47b99656636a033e44b82a542"}
Oct 30 06:00:06.702787 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.702752    5250 scope.go:122] "RemoveContainer" containerID="175685b0e3ec22cb1bc3b46fc2db1723da6826e626d94fa53316094a45077b9c"
Oct 30 06:00:06.702972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.702945    5250 scope.go:122] "RemoveContainer" containerID="047993f617dd83273f7f06e7a3a27de6d35cd5c47b99656636a033e44b82a542"
Oct 30 06:00:06.703347 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:06.703320    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr_odh-ai-gateway-infra(50224823-84da-46b1-9602-3cf83a8e7c27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" podUID="50224823-84da-46b1-9602-3cf83a8e7c27"
Oct 30 06:00:06.706754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.706734    5250 scope.go:122] "RemoveContainer" containerID="3f13dd03925560145f7ddd5bc10ca11fe68ca7ced403e50a8e06fac5f81b0852"
Oct 30 06:00:06.707091 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:06.707067    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf_odh-ai-gateway-infra(5df893bc-ced3-4c63-a549-74f58bcd0423)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423"
Oct 30 06:00:06.709707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.709679    5250 scope.go:122] "RemoveContainer" containerID="967adc9ee145b7d799a4263ea2681fa5eee744abbf062299d5ca2d6df8348ec8"
Oct 30 06:00:06.710046 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:06.710023    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk_odh-ai-gateway-infra(afaa4574-c8d1-4548-a2c7-42924f05c770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770"
Oct 30 06:00:06.711852 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.711830    5250 scope.go:122] "RemoveContainer" containerID="8e398729d863a82ef6b35af1ea08654fd80cae8e5a2bc356c13014da20a1c903"
Oct 30 06:00:06.712229 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:06.712162    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6_odh-ai-gateway-infra(ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04"
Oct 30 06:00:06.713972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.713952    5250 scope.go:122] "RemoveContainer" containerID="42e313e08754dafc6a538783853d475645a360abe386bf1ffc69af2ac36e3808"
Oct 30 06:00:06.714257 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:06.714225    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq_odh-ai-gateway-infra(158297dd-1b28-494d-a338-9305cbc7a461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" podUID="158297dd-1b28-494d-a338-9305cbc7a461"
Oct 30 06:00:06.716276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.716254    5250 scope.go:122] "RemoveContainer" containerID="ce74a63626523b98eba7a626e6fcccc5684162c3173271a7d30519679357bbe8"
Oct 30 06:00:06.716548 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:06.716530    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx_odh-ai-gateway-infra(95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c"
Oct 30 06:00:06.806587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.806476    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:06.987111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:06.987062    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:07.563207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:07.556599    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 06:00:07.563207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:07.556662    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 06:00:07.636213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:07.627859    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:07.827931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:07.827841    5250 scope.go:122] "RemoveContainer" containerID="047993f617dd83273f7f06e7a3a27de6d35cd5c47b99656636a033e44b82a542"
Oct 30 06:00:07.828281 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:07.828247    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr_odh-ai-gateway-infra(50224823-84da-46b1-9602-3cf83a8e7c27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" podUID="50224823-84da-46b1-9602-3cf83a8e7c27"
Oct 30 06:00:08.306846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:08.306818    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:08.559307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:08.559207    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:08.559735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:08.559711    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:08.560387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:08.560368    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:00:09.132467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:09.132359    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:10.141420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:10.141381    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 06:00:10.141842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:10.141467    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 06:00:11.524536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:11.524501    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:13.569517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:13.569479    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:13.804091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:13.804050    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:14.639497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:14.639465    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:00:14.645355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:14.645327    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:00:15.278729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:15.278702    5250 scope.go:122] "RemoveContainer" containerID="282d32528083694e2dc9f45cb9c89e14822cd753f13a14b6cac1ec0f70750356"
Oct 30 06:00:15.513021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:15.512891    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:15.900595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:15.900561    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" event={"ID":"0d2d60ac-6588-46c3-a44a-3f381e576175","Type":"ContainerStarted","Data":"c63c1355559fb681b2815cdf1d3a50d01524886f66e0f388e8875fe7b0bd2134"}
Oct 30 06:00:16.273762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.273735    5250 scope.go:122] "RemoveContainer" containerID="967adc9ee145b7d799a4263ea2681fa5eee744abbf062299d5ca2d6df8348ec8"
Oct 30 06:00:16.273898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.273881    5250 scope.go:122] "RemoveContainer" containerID="ce74a63626523b98eba7a626e6fcccc5684162c3173271a7d30519679357bbe8"
Oct 30 06:00:16.719443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.719418    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:00:16.911251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.911209    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" event={"ID":"afaa4574-c8d1-4548-a2c7-42924f05c770","Type":"ContainerStarted","Data":"fe61838fbff41e7cc05b87200a8d0be01237d8ea0beb8f9f2b1cec7fab1044ce"}
Oct 30 06:00:16.913308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.913285    5250 generic.go:356] "Generic (PLEG): container finished" podID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerID="c63c1355559fb681b2815cdf1d3a50d01524886f66e0f388e8875fe7b0bd2134" exitCode=60
Oct 30 06:00:16.913420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.913355    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" event={"ID":"0d2d60ac-6588-46c3-a44a-3f381e576175","Type":"ContainerDied","Data":"c63c1355559fb681b2815cdf1d3a50d01524886f66e0f388e8875fe7b0bd2134"}
Oct 30 06:00:16.913420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.913404    5250 scope.go:122] "RemoveContainer" containerID="282d32528083694e2dc9f45cb9c89e14822cd753f13a14b6cac1ec0f70750356"
Oct 30 06:00:16.913758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.913737    5250 scope.go:122] "RemoveContainer" containerID="c63c1355559fb681b2815cdf1d3a50d01524886f66e0f388e8875fe7b0bd2134"
Oct 30 06:00:16.914105 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:16.914083    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6_odh-ai-gateway-infra(0d2d60ac-6588-46c3-a44a-3f381e576175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175"
Oct 30 06:00:16.915437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.915414    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" event={"ID":"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c","Type":"ContainerStarted","Data":"7935703ae97974891c6e2e35ea0c3ad457ee89cbed0167e48b6e3a228d4d720c"}
Oct 30 06:00:16.959359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:16.959322    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"]
Oct 30 06:00:17.273783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.273710    5250 scope.go:122] "RemoveContainer" containerID="3f13dd03925560145f7ddd5bc10ca11fe68ca7ced403e50a8e06fac5f81b0852"
Oct 30 06:00:17.299260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.299224    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:17.599647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.599619    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:17.768270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.768241    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:17.768420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.768326    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"]
Oct 30 06:00:17.773680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.773663    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:17.790677 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.790656    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8de592\""
Oct 30 06:00:17.791085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.791069    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"]
Oct 30 06:00:17.792890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.792868    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8de592-dockercfg-84vwf\""
Oct 30 06:00:17.830351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.830327    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:00:17.850826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.850796    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fad536f-7c39-4463-a60d-e34aae5fc8de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj\" (UID: \"8fad536f-7c39-4463-a60d-e34aae5fc8de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:17.851060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.851035    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjd6\" (UniqueName: \"kubernetes.io/projected/8fad536f-7c39-4463-a60d-e34aae5fc8de-kube-api-access-qzjd6\") pod \"payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj\" (UID: \"8fad536f-7c39-4463-a60d-e34aae5fc8de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:17.873698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.873641    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"]
Oct 30 06:00:17.879831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:17.879802    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:18.084302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.061641    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"]
Oct 30 06:00:18.107233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.104042    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fad536f-7c39-4463-a60d-e34aae5fc8de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj\" (UID: \"8fad536f-7c39-4463-a60d-e34aae5fc8de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:18.107233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.104751    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjd6\" (UniqueName: \"kubernetes.io/projected/8fad536f-7c39-4463-a60d-e34aae5fc8de-kube-api-access-qzjd6\") pod \"payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj\" (UID: \"8fad536f-7c39-4463-a60d-e34aae5fc8de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:18.123138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.122977    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fad536f-7c39-4463-a60d-e34aae5fc8de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj\" (UID: \"8fad536f-7c39-4463-a60d-e34aae5fc8de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:18.158912 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.158810    5250 generic.go:356] "Generic (PLEG): container finished" podID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerID="7935703ae97974891c6e2e35ea0c3ad457ee89cbed0167e48b6e3a228d4d720c" exitCode=60
Oct 30 06:00:18.159711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.159420    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" event={"ID":"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c","Type":"ContainerDied","Data":"7935703ae97974891c6e2e35ea0c3ad457ee89cbed0167e48b6e3a228d4d720c"}
Oct 30 06:00:18.159711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.159542    5250 scope.go:122] "RemoveContainer" containerID="ce74a63626523b98eba7a626e6fcccc5684162c3173271a7d30519679357bbe8"
Oct 30 06:00:18.160425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.160221    5250 scope.go:122] "RemoveContainer" containerID="7935703ae97974891c6e2e35ea0c3ad457ee89cbed0167e48b6e3a228d4d720c"
Oct 30 06:00:18.160890 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:18.160851    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx_odh-ai-gateway-infra(95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c"
Oct 30 06:00:18.166607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.166562    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" event={"ID":"5df893bc-ced3-4c63-a549-74f58bcd0423","Type":"ContainerStarted","Data":"7d5ed1d90b8964455b339e47a1f9940d592fe9d43fef39728ab3df0ec256499a"}
Oct 30 06:00:18.171332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.171301    5250 generic.go:356] "Generic (PLEG): container finished" podID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerID="fe61838fbff41e7cc05b87200a8d0be01237d8ea0beb8f9f2b1cec7fab1044ce" exitCode=60
Oct 30 06:00:18.171537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.171461    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" event={"ID":"afaa4574-c8d1-4548-a2c7-42924f05c770","Type":"ContainerDied","Data":"fe61838fbff41e7cc05b87200a8d0be01237d8ea0beb8f9f2b1cec7fab1044ce"}
Oct 30 06:00:18.172590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.172516    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjd6\" (UniqueName: \"kubernetes.io/projected/8fad536f-7c39-4463-a60d-e34aae5fc8de-kube-api-access-qzjd6\") pod \"payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj\" (UID: \"8fad536f-7c39-4463-a60d-e34aae5fc8de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:18.175502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.175480    5250 scope.go:122] "RemoveContainer" containerID="fe61838fbff41e7cc05b87200a8d0be01237d8ea0beb8f9f2b1cec7fab1044ce"
Oct 30 06:00:18.176060 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:18.176034    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk_odh-ai-gateway-infra(afaa4574-c8d1-4548-a2c7-42924f05c770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770"
Oct 30 06:00:18.181093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.181071    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"]
Oct 30 06:00:18.195933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.195758    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"]
Oct 30 06:00:18.199425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.199402    5250 scope.go:122] "RemoveContainer" containerID="967adc9ee145b7d799a4263ea2681fa5eee744abbf062299d5ca2d6df8348ec8"
Oct 30 06:00:18.224120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.224082    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52574608-d7bd-49b1-a833-e81372b261c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd\" (UID: \"52574608-d7bd-49b1-a833-e81372b261c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:18.224395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.224363    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljk6t\" (UniqueName: \"kubernetes.io/projected/52574608-d7bd-49b1-a833-e81372b261c0-kube-api-access-ljk6t\") pod \"payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd\" (UID: \"52574608-d7bd-49b1-a833-e81372b261c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:18.274291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.274240    5250 scope.go:122] "RemoveContainer" containerID="8e398729d863a82ef6b35af1ea08654fd80cae8e5a2bc356c13014da20a1c903"
Oct 30 06:00:18.275738 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.275719    5250 scope.go:122] "RemoveContainer" containerID="42e313e08754dafc6a538783853d475645a360abe386bf1ffc69af2ac36e3808"
Oct 30 06:00:18.328817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.328354    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljk6t\" (UniqueName: \"kubernetes.io/projected/52574608-d7bd-49b1-a833-e81372b261c0-kube-api-access-ljk6t\") pod \"payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd\" (UID: \"52574608-d7bd-49b1-a833-e81372b261c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:18.331599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.331563    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52574608-d7bd-49b1-a833-e81372b261c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd\" (UID: \"52574608-d7bd-49b1-a833-e81372b261c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:18.332438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.332412    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52574608-d7bd-49b1-a833-e81372b261c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd\" (UID: \"52574608-d7bd-49b1-a833-e81372b261c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:18.353284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.353033    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljk6t\" (UniqueName: \"kubernetes.io/projected/52574608-d7bd-49b1-a833-e81372b261c0-kube-api-access-ljk6t\") pod \"payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd\" (UID: \"52574608-d7bd-49b1-a833-e81372b261c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:18.389983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.389959    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:18.436788 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.436446    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:18.462902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.462581    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:18.534734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.534694    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2d60ac-6588-46c3-a44a-3f381e576175-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d60ac-6588-46c3-a44a-3f381e576175-maas-api-service-ca\") pod \"0d2d60ac-6588-46c3-a44a-3f381e576175\" (UID: \"0d2d60ac-6588-46c3-a44a-3f381e576175\") "
Oct 30 06:00:18.534899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.534807    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2d60ac-6588-46c3-a44a-3f381e576175-kube-api-access-txtp6\" (UniqueName: \"kubernetes.io/projected/0d2d60ac-6588-46c3-a44a-3f381e576175-kube-api-access-txtp6\") pod \"0d2d60ac-6588-46c3-a44a-3f381e576175\" (UID: \"0d2d60ac-6588-46c3-a44a-3f381e576175\") "
Oct 30 06:00:18.535426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.535396    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2d60ac-6588-46c3-a44a-3f381e576175-maas-api-service-ca" pod "0d2d60ac-6588-46c3-a44a-3f381e576175" (UID: "0d2d60ac-6588-46c3-a44a-3f381e576175"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:18.557240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.557204    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2d60ac-6588-46c3-a44a-3f381e576175-kube-api-access-txtp6" pod "0d2d60ac-6588-46c3-a44a-3f381e576175" (UID: "0d2d60ac-6588-46c3-a44a-3f381e576175"). InnerVolumeSpecName "kube-api-access-txtp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:18.637053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.637017    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d60ac-6588-46c3-a44a-3f381e576175-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:18.637053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.637052    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txtp6\" (UniqueName: \"kubernetes.io/projected/0d2d60ac-6588-46c3-a44a-3f381e576175-kube-api-access-txtp6\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:18.704760 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.704726    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"]
Oct 30 06:00:18.705141 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:18.705115    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fad536f_7c39_4463_a60d_e34aae5fc8de.slice/crio-e773bae755fa32875181aa2afe8d7d2d6a632235cd8fef786d3559a500b0436c WatchSource:0}: Error finding container e773bae755fa32875181aa2afe8d7d2d6a632235cd8fef786d3559a500b0436c: Status 404 returned error can't find the container with id e773bae755fa32875181aa2afe8d7d2d6a632235cd8fef786d3559a500b0436c
Oct 30 06:00:18.866685 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:18.866650    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52574608_d7bd_49b1_a833_e81372b261c0.slice/crio-5c8ec43ae385f73e29f9c09cd78800426352981cf0c5ced66ff15a2a6ced9158 WatchSource:0}: Error finding container 5c8ec43ae385f73e29f9c09cd78800426352981cf0c5ced66ff15a2a6ced9158: Status 404 returned error can't find the container with id 5c8ec43ae385f73e29f9c09cd78800426352981cf0c5ced66ff15a2a6ced9158
Oct 30 06:00:18.866685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:18.866658    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"]
Oct 30 06:00:19.187950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.187908    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6" event={"ID":"0d2d60ac-6588-46c3-a44a-3f381e576175","Type":"ContainerDied","Data":"a6b1c121c00208aa31796902f055d77d9242b96522d3dd9175da0de5ccad11d8"}
Oct 30 06:00:19.188467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.187989    5250 scope.go:122] "RemoveContainer" containerID="c63c1355559fb681b2815cdf1d3a50d01524886f66e0f388e8875fe7b0bd2134"
Oct 30 06:00:19.188467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.187924    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"
Oct 30 06:00:19.190533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.190499    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" event={"ID":"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04","Type":"ContainerStarted","Data":"7c16b7a708d4827df5c9471e7a7a027303d2c01177ceacb9d76981eedd9fa1bb"}
Oct 30 06:00:19.193128 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.193098    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" event={"ID":"158297dd-1b28-494d-a338-9305cbc7a461","Type":"ContainerStarted","Data":"8664c0dcdee4dad4a477a8a439dab9532d5b5320c086f5a6d94e887aa3734ac2"}
Oct 30 06:00:19.197357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.197331    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj" event={"ID":"8fad536f-7c39-4463-a60d-e34aae5fc8de","Type":"ContainerStarted","Data":"63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce"}
Oct 30 06:00:19.197528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.197378    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj" event={"ID":"8fad536f-7c39-4463-a60d-e34aae5fc8de","Type":"ContainerStarted","Data":"e773bae755fa32875181aa2afe8d7d2d6a632235cd8fef786d3559a500b0436c"}
Oct 30 06:00:19.197528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.197433    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:19.201679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.201640    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd" event={"ID":"52574608-d7bd-49b1-a833-e81372b261c0","Type":"ContainerStarted","Data":"8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413"}
Oct 30 06:00:19.201679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.201674    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd" event={"ID":"52574608-d7bd-49b1-a833-e81372b261c0","Type":"ContainerStarted","Data":"5c8ec43ae385f73e29f9c09cd78800426352981cf0c5ced66ff15a2a6ced9158"}
Oct 30 06:00:19.202434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.202418    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:19.204551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.204530    5250 generic.go:356] "Generic (PLEG): container finished" podID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerID="7d5ed1d90b8964455b339e47a1f9940d592fe9d43fef39728ab3df0ec256499a" exitCode=60
Oct 30 06:00:19.204660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.204595    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" event={"ID":"5df893bc-ced3-4c63-a549-74f58bcd0423","Type":"ContainerDied","Data":"7d5ed1d90b8964455b339e47a1f9940d592fe9d43fef39728ab3df0ec256499a"}
Oct 30 06:00:19.204660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.204633    5250 scope.go:122] "RemoveContainer" containerID="3f13dd03925560145f7ddd5bc10ca11fe68ca7ced403e50a8e06fac5f81b0852"
Oct 30 06:00:19.204991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.204972    5250 scope.go:122] "RemoveContainer" containerID="7d5ed1d90b8964455b339e47a1f9940d592fe9d43fef39728ab3df0ec256499a"
Oct 30 06:00:19.205322 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:19.205295    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf_odh-ai-gateway-infra(5df893bc-ced3-4c63-a549-74f58bcd0423)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423"
Oct 30 06:00:19.278267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.278238    5250 scope.go:122] "RemoveContainer" containerID="047993f617dd83273f7f06e7a3a27de6d35cd5c47b99656636a033e44b82a542"
Oct 30 06:00:19.278626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.278603    5250 scope.go:122] "RemoveContainer" containerID="3646c9d40a4a3fada2f26db048c21c32316ce5bcffd8960b0e7f1d650f2637ff"
Oct 30 06:00:19.301877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.301843    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:19.335220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.333246    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"]
Oct 30 06:00:19.359824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.359772    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj" podStartSLOduration=2.359751336 podStartE2EDuration="2.359751336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:19.35392509 +0000 UTC m=+1734.184962308" watchObservedRunningTime="2026-10-30 06:00:19.359751336 +0000 UTC m=+1734.190788554"
Oct 30 06:00:19.433676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.433547    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:19.465029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.464926    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-kube-api-access-sjpj8\" (UniqueName: \"kubernetes.io/projected/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-kube-api-access-sjpj8\") pod \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\" (UID: \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\") "
Oct 30 06:00:19.466313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.466286    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-maas-api-service-ca\") pod \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\" (UID: \"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c\") "
Oct 30 06:00:19.466936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.466905    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-maas-api-service-ca" pod "95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" (UID: "95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:19.470746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.468057    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:19.477946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.477906    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-kube-api-access-sjpj8" pod "95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" (UID: "95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c"). InnerVolumeSpecName "kube-api-access-sjpj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:19.495730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.495673    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd" podStartSLOduration=2.495656431 podStartE2EDuration="2.495656431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:19.469116554 +0000 UTC m=+1734.300153772" watchObservedRunningTime="2026-10-30 06:00:19.495656431 +0000 UTC m=+1734.326693692"
Oct 30 06:00:19.527976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.527883    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"]
Oct 30 06:00:19.548145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.548089    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889000-vg4x6"]
Oct 30 06:00:19.573363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.573331    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjpj8\" (UniqueName: \"kubernetes.io/projected/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c-kube-api-access-sjpj8\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:19.592106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.592073    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:19.674064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.674035    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afaa4574-c8d1-4548-a2c7-42924f05c770-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afaa4574-c8d1-4548-a2c7-42924f05c770-maas-api-service-ca\") pod \"afaa4574-c8d1-4548-a2c7-42924f05c770\" (UID: \"afaa4574-c8d1-4548-a2c7-42924f05c770\") "
Oct 30 06:00:19.674467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.674423    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afaa4574-c8d1-4548-a2c7-42924f05c770-maas-api-service-ca" pod "afaa4574-c8d1-4548-a2c7-42924f05c770" (UID: "afaa4574-c8d1-4548-a2c7-42924f05c770"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:19.674625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.674471    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afaa4574-c8d1-4548-a2c7-42924f05c770-kube-api-access-gzbjg\" (UniqueName: \"kubernetes.io/projected/afaa4574-c8d1-4548-a2c7-42924f05c770-kube-api-access-gzbjg\") pod \"afaa4574-c8d1-4548-a2c7-42924f05c770\" (UID: \"afaa4574-c8d1-4548-a2c7-42924f05c770\") "
Oct 30 06:00:19.675475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.675162    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afaa4574-c8d1-4548-a2c7-42924f05c770-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:19.688736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.688691    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afaa4574-c8d1-4548-a2c7-42924f05c770-kube-api-access-gzbjg" pod "afaa4574-c8d1-4548-a2c7-42924f05c770" (UID: "afaa4574-c8d1-4548-a2c7-42924f05c770"). InnerVolumeSpecName "kube-api-access-gzbjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:19.777042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:19.777007    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbjg\" (UniqueName: \"kubernetes.io/projected/afaa4574-c8d1-4548-a2c7-42924f05c770-kube-api-access-gzbjg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:20.216920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.216887    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"
Oct 30 06:00:20.216920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.216901    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk" event={"ID":"afaa4574-c8d1-4548-a2c7-42924f05c770","Type":"ContainerDied","Data":"bd27a1b64d61beea396abc96c06e3c57629171dbdfa6b4bd95ecb00c94bed053"}
Oct 30 06:00:20.217500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.217008    5250 scope.go:122] "RemoveContainer" containerID="fe61838fbff41e7cc05b87200a8d0be01237d8ea0beb8f9f2b1cec7fab1044ce"
Oct 30 06:00:20.224398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.224367    5250 generic.go:356] "Generic (PLEG): container finished" podID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerID="7c16b7a708d4827df5c9471e7a7a027303d2c01177ceacb9d76981eedd9fa1bb" exitCode=60
Oct 30 06:00:20.224547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.224451    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" event={"ID":"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04","Type":"ContainerDied","Data":"7c16b7a708d4827df5c9471e7a7a027303d2c01177ceacb9d76981eedd9fa1bb"}
Oct 30 06:00:20.224961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.224937    5250 scope.go:122] "RemoveContainer" containerID="7c16b7a708d4827df5c9471e7a7a027303d2c01177ceacb9d76981eedd9fa1bb"
Oct 30 06:00:20.225389 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:20.225349    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6_odh-ai-gateway-infra(ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04"
Oct 30 06:00:20.227004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.226986    5250 generic.go:356] "Generic (PLEG): container finished" podID="158297dd-1b28-494d-a338-9305cbc7a461" containerID="8664c0dcdee4dad4a477a8a439dab9532d5b5320c086f5a6d94e887aa3734ac2" exitCode=60
Oct 30 06:00:20.227097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.227059    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" event={"ID":"158297dd-1b28-494d-a338-9305cbc7a461","Type":"ContainerDied","Data":"8664c0dcdee4dad4a477a8a439dab9532d5b5320c086f5a6d94e887aa3734ac2"}
Oct 30 06:00:20.227529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.227509    5250 scope.go:122] "RemoveContainer" containerID="8664c0dcdee4dad4a477a8a439dab9532d5b5320c086f5a6d94e887aa3734ac2"
Oct 30 06:00:20.227910 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:20.227888    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq_odh-ai-gateway-infra(158297dd-1b28-494d-a338-9305cbc7a461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" podUID="158297dd-1b28-494d-a338-9305cbc7a461"
Oct 30 06:00:20.228640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.228626    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"
Oct 30 06:00:20.228706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.228673    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx" event={"ID":"95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c","Type":"ContainerDied","Data":"e00ad7e788c8f78c9bcd57f4cba845fa60433057c534446a358e4915132e4c33"}
Oct 30 06:00:20.231378 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.230386    5250 scope.go:122] "RemoveContainer" containerID="8e398729d863a82ef6b35af1ea08654fd80cae8e5a2bc356c13014da20a1c903"
Oct 30 06:00:20.231378 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.230795    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" event={"ID":"34b5631a-0e49-442c-bcee-c3324d909233","Type":"ContainerStarted","Data":"537aae6f492206cc6b7680039316e215c27fb6a79cdbc92cce50809cc81afc4f"}
Oct 30 06:00:20.234102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.234073    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" event={"ID":"50224823-84da-46b1-9602-3cf83a8e7c27","Type":"ContainerStarted","Data":"bb9db61f6708307d5c51ee430f7ca6eedf321dc7acfbeed214a7e313cb52f1b0"}
Oct 30 06:00:20.247399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.247376    5250 scope.go:122] "RemoveContainer" containerID="42e313e08754dafc6a538783853d475645a360abe386bf1ffc69af2ac36e3808"
Oct 30 06:00:20.266471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.266448    5250 scope.go:122] "RemoveContainer" containerID="7935703ae97974891c6e2e35ea0c3ad457ee89cbed0167e48b6e3a228d4d720c"
Oct 30 06:00:20.308064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.308032    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"]
Oct 30 06:00:20.370579 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.370547    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"]
Oct 30 06:00:20.377286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.377257    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889000-zwxqk"]
Oct 30 06:00:20.412056 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.412023    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"]
Oct 30 06:00:20.422165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.422138    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:20.489410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.489380    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df893bc-ced3-4c63-a549-74f58bcd0423-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df893bc-ced3-4c63-a549-74f58bcd0423-maas-api-service-ca\") pod \"5df893bc-ced3-4c63-a549-74f58bcd0423\" (UID: \"5df893bc-ced3-4c63-a549-74f58bcd0423\") "
Oct 30 06:00:20.489535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.489497    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df893bc-ced3-4c63-a549-74f58bcd0423-kube-api-access-65hr6\" (UniqueName: \"kubernetes.io/projected/5df893bc-ced3-4c63-a549-74f58bcd0423-kube-api-access-65hr6\") pod \"5df893bc-ced3-4c63-a549-74f58bcd0423\" (UID: \"5df893bc-ced3-4c63-a549-74f58bcd0423\") "
Oct 30 06:00:20.489733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.489707    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df893bc-ced3-4c63-a549-74f58bcd0423-maas-api-service-ca" pod "5df893bc-ced3-4c63-a549-74f58bcd0423" (UID: "5df893bc-ced3-4c63-a549-74f58bcd0423"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:20.490329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.490308    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df893bc-ced3-4c63-a549-74f58bcd0423-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:20.491783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.491766    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df893bc-ced3-4c63-a549-74f58bcd0423-kube-api-access-65hr6" pod "5df893bc-ced3-4c63-a549-74f58bcd0423" (UID: "5df893bc-ced3-4c63-a549-74f58bcd0423"). InnerVolumeSpecName "kube-api-access-65hr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:20.561295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.561258    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"]
Oct 30 06:00:20.563323 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.563300    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39147c-29889000-mf6rx"]
Oct 30 06:00:20.592151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:20.592124    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65hr6\" (UniqueName: \"kubernetes.io/projected/5df893bc-ced3-4c63-a549-74f58bcd0423-kube-api-access-65hr6\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:21.048434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.048020    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:21.246986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.246954    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"
Oct 30 06:00:21.247576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.246954    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf" event={"ID":"5df893bc-ced3-4c63-a549-74f58bcd0423","Type":"ContainerDied","Data":"ca21621729d5e222826a259b693fa6af092c62cd69c669407909966be6dae5cb"}
Oct 30 06:00:21.247576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.247104    5250 scope.go:122] "RemoveContainer" containerID="7d5ed1d90b8964455b339e47a1f9940d592fe9d43fef39728ab3df0ec256499a"
Oct 30 06:00:21.255983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.255955    5250 generic.go:356] "Generic (PLEG): container finished" podID="34b5631a-0e49-442c-bcee-c3324d909233" containerID="537aae6f492206cc6b7680039316e215c27fb6a79cdbc92cce50809cc81afc4f" exitCode=60
Oct 30 06:00:21.256108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.256030    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" event={"ID":"34b5631a-0e49-442c-bcee-c3324d909233","Type":"ContainerDied","Data":"537aae6f492206cc6b7680039316e215c27fb6a79cdbc92cce50809cc81afc4f"}
Oct 30 06:00:21.256587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.256564    5250 scope.go:122] "RemoveContainer" containerID="537aae6f492206cc6b7680039316e215c27fb6a79cdbc92cce50809cc81afc4f"
Oct 30 06:00:21.257172 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:21.256985    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d_odh-ai-gateway-infra(34b5631a-0e49-442c-bcee-c3324d909233)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" podUID="34b5631a-0e49-442c-bcee-c3324d909233"
Oct 30 06:00:21.259454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.259433    5250 generic.go:356] "Generic (PLEG): container finished" podID="50224823-84da-46b1-9602-3cf83a8e7c27" containerID="bb9db61f6708307d5c51ee430f7ca6eedf321dc7acfbeed214a7e313cb52f1b0" exitCode=60
Oct 30 06:00:21.259680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.259657    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" event={"ID":"50224823-84da-46b1-9602-3cf83a8e7c27","Type":"ContainerDied","Data":"bb9db61f6708307d5c51ee430f7ca6eedf321dc7acfbeed214a7e313cb52f1b0"}
Oct 30 06:00:21.260006 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.259988    5250 scope.go:122] "RemoveContainer" containerID="bb9db61f6708307d5c51ee430f7ca6eedf321dc7acfbeed214a7e313cb52f1b0"
Oct 30 06:00:21.260338 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:21.260319    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr_odh-ai-gateway-infra(50224823-84da-46b1-9602-3cf83a8e7c27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" podUID="50224823-84da-46b1-9602-3cf83a8e7c27"
Oct 30 06:00:21.262465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.262447    5250 scope.go:122] "RemoveContainer" containerID="3646c9d40a4a3fada2f26db048c21c32316ce5bcffd8960b0e7f1d650f2637ff"
Oct 30 06:00:21.282050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.282021    5250 scope.go:122] "RemoveContainer" containerID="047993f617dd83273f7f06e7a3a27de6d35cd5c47b99656636a033e44b82a542"
Oct 30 06:00:21.287460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.287432    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" path="/var/lib/kubelet/pods/0d2d60ac-6588-46c3-a44a-3f381e576175/volumes"
Oct 30 06:00:21.288233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.288164    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" path="/var/lib/kubelet/pods/95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c/volumes"
Oct 30 06:00:21.288791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.288759    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" path="/var/lib/kubelet/pods/afaa4574-c8d1-4548-a2c7-42924f05c770/volumes"
Oct 30 06:00:21.345283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.345252    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"]
Oct 30 06:00:21.463213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.463169    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:21.484837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.484803    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"]
Oct 30 06:00:21.498374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.498347    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"]
Oct 30 06:00:21.502531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.502505    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e625-29889000-nfktf"]
Oct 30 06:00:21.510404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.510374    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-maas-api-service-ca\") pod \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\" (UID: \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\") "
Oct 30 06:00:21.510535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.510472    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-kube-api-access-f97f4\" (UniqueName: \"kubernetes.io/projected/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-kube-api-access-f97f4\") pod \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\" (UID: \"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04\") "
Oct 30 06:00:21.510843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.510816    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-maas-api-service-ca" pod "ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" (UID: "ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:21.511583 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.511555    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:21.512860 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.512840    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-kube-api-access-f97f4" pod "ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" (UID: "ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04"). InnerVolumeSpecName "kube-api-access-f97f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:21.514048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.514026    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:21.613039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.613010    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158297dd-1b28-494d-a338-9305cbc7a461-kube-api-access-5ttmm\" (UniqueName: \"kubernetes.io/projected/158297dd-1b28-494d-a338-9305cbc7a461-kube-api-access-5ttmm\") pod \"158297dd-1b28-494d-a338-9305cbc7a461\" (UID: \"158297dd-1b28-494d-a338-9305cbc7a461\") "
Oct 30 06:00:21.613349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.613333    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158297dd-1b28-494d-a338-9305cbc7a461-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158297dd-1b28-494d-a338-9305cbc7a461-maas-api-service-ca\") pod \"158297dd-1b28-494d-a338-9305cbc7a461\" (UID: \"158297dd-1b28-494d-a338-9305cbc7a461\") "
Oct 30 06:00:21.613773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.613740    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158297dd-1b28-494d-a338-9305cbc7a461-maas-api-service-ca" pod "158297dd-1b28-494d-a338-9305cbc7a461" (UID: "158297dd-1b28-494d-a338-9305cbc7a461"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:21.613925 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.613812    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97f4\" (UniqueName: \"kubernetes.io/projected/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04-kube-api-access-f97f4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:21.615558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.615532    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158297dd-1b28-494d-a338-9305cbc7a461-kube-api-access-5ttmm" pod "158297dd-1b28-494d-a338-9305cbc7a461" (UID: "158297dd-1b28-494d-a338-9305cbc7a461"). InnerVolumeSpecName "kube-api-access-5ttmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:21.715533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.715504    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158297dd-1b28-494d-a338-9305cbc7a461-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:21.715533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:21.715532    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ttmm\" (UniqueName: \"kubernetes.io/projected/158297dd-1b28-494d-a338-9305cbc7a461-kube-api-access-5ttmm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:22.268268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.268236    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq" event={"ID":"158297dd-1b28-494d-a338-9305cbc7a461","Type":"ContainerDied","Data":"59145f88060195a4ae3c151fe07b8a2ed38e1ab9a13bb20cb149843a184d56d8"}
Oct 30 06:00:22.268268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.268251    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"
Oct 30 06:00:22.268790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.268304    5250 scope.go:122] "RemoveContainer" containerID="8664c0dcdee4dad4a477a8a439dab9532d5b5320c086f5a6d94e887aa3734ac2"
Oct 30 06:00:22.275743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.275701    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6" event={"ID":"ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04","Type":"ContainerDied","Data":"2dffd9fb48c27d06e177a16b1174b31346c1b3a6dd6ac983a4bbd924a5401c59"}
Oct 30 06:00:22.275856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.275717    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"
Oct 30 06:00:22.289540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.289277    5250 scope.go:122] "RemoveContainer" containerID="7c16b7a708d4827df5c9471e7a7a027303d2c01177ceacb9d76981eedd9fa1bb"
Oct 30 06:00:22.378109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.378080    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"]
Oct 30 06:00:22.384431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.384401    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889000-t49fq"]
Oct 30 06:00:22.457542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.457510    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"]
Oct 30 06:00:22.468828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.468796    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:22.469437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.469418    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889000-svqh6"]
Oct 30 06:00:22.497436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.497406    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:22.526745 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.526655    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b5631a-0e49-442c-bcee-c3324d909233-kube-api-access-sld8t\" (UniqueName: \"kubernetes.io/projected/34b5631a-0e49-442c-bcee-c3324d909233-kube-api-access-sld8t\") pod \"34b5631a-0e49-442c-bcee-c3324d909233\" (UID: \"34b5631a-0e49-442c-bcee-c3324d909233\") "
Oct 30 06:00:22.526899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.526827    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b5631a-0e49-442c-bcee-c3324d909233-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5631a-0e49-442c-bcee-c3324d909233-maas-api-service-ca\") pod \"34b5631a-0e49-442c-bcee-c3324d909233\" (UID: \"34b5631a-0e49-442c-bcee-c3324d909233\") "
Oct 30 06:00:22.527675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.527473    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b5631a-0e49-442c-bcee-c3324d909233-maas-api-service-ca" pod "34b5631a-0e49-442c-bcee-c3324d909233" (UID: "34b5631a-0e49-442c-bcee-c3324d909233"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:22.527675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.527639    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5631a-0e49-442c-bcee-c3324d909233-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:22.529301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.529268    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b5631a-0e49-442c-bcee-c3324d909233-kube-api-access-sld8t" pod "34b5631a-0e49-442c-bcee-c3324d909233" (UID: "34b5631a-0e49-442c-bcee-c3324d909233"). InnerVolumeSpecName "kube-api-access-sld8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:22.629232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.629190    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50224823-84da-46b1-9602-3cf83a8e7c27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50224823-84da-46b1-9602-3cf83a8e7c27-maas-api-service-ca\") pod \"50224823-84da-46b1-9602-3cf83a8e7c27\" (UID: \"50224823-84da-46b1-9602-3cf83a8e7c27\") "
Oct 30 06:00:22.629516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.629486    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50224823-84da-46b1-9602-3cf83a8e7c27-kube-api-access-jzhtc\" (UniqueName: \"kubernetes.io/projected/50224823-84da-46b1-9602-3cf83a8e7c27-kube-api-access-jzhtc\") pod \"50224823-84da-46b1-9602-3cf83a8e7c27\" (UID: \"50224823-84da-46b1-9602-3cf83a8e7c27\") "
Oct 30 06:00:22.629942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.629910    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50224823-84da-46b1-9602-3cf83a8e7c27-maas-api-service-ca" pod "50224823-84da-46b1-9602-3cf83a8e7c27" (UID: "50224823-84da-46b1-9602-3cf83a8e7c27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:22.630382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.630359    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sld8t\" (UniqueName: \"kubernetes.io/projected/34b5631a-0e49-442c-bcee-c3324d909233-kube-api-access-sld8t\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:22.630462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.630388    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50224823-84da-46b1-9602-3cf83a8e7c27-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:22.632049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.632021    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50224823-84da-46b1-9602-3cf83a8e7c27-kube-api-access-jzhtc" pod "50224823-84da-46b1-9602-3cf83a8e7c27" (UID: "50224823-84da-46b1-9602-3cf83a8e7c27"). InnerVolumeSpecName "kube-api-access-jzhtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:22.685101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.683489    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:22.732403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:22.732368    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhtc\" (UniqueName: \"kubernetes.io/projected/50224823-84da-46b1-9602-3cf83a8e7c27-kube-api-access-jzhtc\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:23.284079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.283980    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158297dd-1b28-494d-a338-9305cbc7a461" path="/var/lib/kubelet/pods/158297dd-1b28-494d-a338-9305cbc7a461/volumes"
Oct 30 06:00:23.284724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.284702    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" path="/var/lib/kubelet/pods/5df893bc-ced3-4c63-a549-74f58bcd0423/volumes"
Oct 30 06:00:23.285221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.285166    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" path="/var/lib/kubelet/pods/ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04/volumes"
Oct 30 06:00:23.287750 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.287727    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d" event={"ID":"34b5631a-0e49-442c-bcee-c3324d909233","Type":"ContainerDied","Data":"9294f214f41e7c2327328ae705dbef97c779d6baee9e2d920f333eecfe15d17c"}
Oct 30 06:00:23.287868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.287736    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"
Oct 30 06:00:23.287934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.287780    5250 scope.go:122] "RemoveContainer" containerID="537aae6f492206cc6b7680039316e215c27fb6a79cdbc92cce50809cc81afc4f"
Oct 30 06:00:23.289806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.289750    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr" event={"ID":"50224823-84da-46b1-9602-3cf83a8e7c27","Type":"ContainerDied","Data":"00d7a005f0506cfec2d90d584e80ff3bf166f0dbea95e22e19f286e6d2549de6"}
Oct 30 06:00:23.289806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.289780    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"
Oct 30 06:00:23.301661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.301643    5250 scope.go:122] "RemoveContainer" containerID="bb9db61f6708307d5c51ee430f7ca6eedf321dc7acfbeed214a7e313cb52f1b0"
Oct 30 06:00:23.355957 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.355926    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"]
Oct 30 06:00:23.375499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.369759    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889000-htbrr"]
Oct 30 06:00:23.391442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.391414    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"]
Oct 30 06:00:23.395129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:23.395106    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889000-rs57d"]
Oct 30 06:00:24.332477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:24.332433    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:25.284297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:25.284262    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b5631a-0e49-442c-bcee-c3324d909233" path="/var/lib/kubelet/pods/34b5631a-0e49-442c-bcee-c3324d909233/volumes"
Oct 30 06:00:25.284791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:25.284776    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" path="/var/lib/kubelet/pods/50224823-84da-46b1-9602-3cf83a8e7c27/volumes"
Oct 30 06:00:25.932902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:25.932869    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:26.084762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:26.084710    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:26.232800 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:26.232717    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:27.538410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:27.538379    5250 scope.go:122] "RemoveContainer" containerID="5878625800ae1b0616e774a44d5f3b5dd80f371cef0b7a2d3e81631a7c8e9d41"
Oct 30 06:00:27.549518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:27.549495    5250 scope.go:122] "RemoveContainer" containerID="bd53117abd727563e62cb6e59bea3311b0796cd9390deda33c9fea337987640a"
Oct 30 06:00:27.946727 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:27.946672    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:28.148287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:28.148243    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:29.820142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:29.820083    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:30.239000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.238969    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:00:30.404054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404020    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"]
Oct 30 06:00:30.404783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404764    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:30.404783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404780    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404789    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404794    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404804    5250 cpu_manager.go:414] "containerMap: removing container" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404809    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404823    5250 cpu_manager.go:414] "containerMap: removing container" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404828    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404837    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404841    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404849    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404855    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404861    5250 cpu_manager.go:414] "containerMap: removing container" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404865    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404876    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404881    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404887    5250 cpu_manager.go:414] "containerMap: removing container" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404892    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404899    5250 cpu_manager.go:414] "containerMap: removing container" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404903    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404911    5250 cpu_manager.go:414] "containerMap: removing container" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404915    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404925    5250 cpu_manager.go:414] "containerMap: removing container" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404929    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404940    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404946    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404955    5250 cpu_manager.go:414] "containerMap: removing container" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404959    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404964    5250 cpu_manager.go:414] "containerMap: removing container" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404968    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:30.404959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404977    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404982    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.404996    5250 cpu_manager.go:414] "containerMap: removing container" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405000    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405008    5250 cpu_manager.go:414] "containerMap: removing container" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405013    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405019    5250 cpu_manager.go:414] "containerMap: removing container" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405023    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405031    5250 cpu_manager.go:414] "containerMap: removing container" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405035    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405133    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405140    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405146    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405153    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405158    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405163    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405168    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405172    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405197    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405202    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405207    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405213    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405218    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405223    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405228    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5a32d8-d14e-4a3e-8fd1-c02aaa6aaa04" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405234    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405240    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="50224823-84da-46b1-9602-3cf83a8e7c27" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405245    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405251    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405256    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b5631a-0e49-442c-bcee-c3324d909233" containerName="cleanup"
Oct 30 06:00:30.406521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.405268    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:30.411524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.411503    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.456200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.456143    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"]
Oct 30 06:00:30.474395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.474364    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.474547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.474469    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.474547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.474529    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.474652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.474592    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdmm\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-kube-api-access-ccdmm\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.474700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.474688    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.474737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.474711    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.474787 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.474753    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872e7a34-3fe4-4ae5-b29b-26848745a69c-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.475003 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.474976    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.475062 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.475023    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576451    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576527    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdmm\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-kube-api-access-ccdmm\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576609    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576634    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576685    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872e7a34-3fe4-4ae5-b29b-26848745a69c-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576850    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576852    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576882    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.576979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576919    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.577194 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.576976    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.577194 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.577059    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.577194 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.577117    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.577370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.577213    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.577458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.577437    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872e7a34-3fe4-4ae5-b29b-26848745a69c-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.579437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.579413    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.579648 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.579632    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.600870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.600843    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.601008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.600947    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdmm\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-kube-api-access-ccdmm\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:30.728980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:30.728940    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:31.062408 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:31.062378    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872e7a34_3fe4_4ae5_b29b_26848745a69c.slice/crio-ba8ae0b94c1835c5cac5d02d55c7f1f24992c1ac544069e17eb56875d4dd2bac WatchSource:0}: Error finding container ba8ae0b94c1835c5cac5d02d55c7f1f24992c1ac544069e17eb56875d4dd2bac: Status 404 returned error can't find the container with id ba8ae0b94c1835c5cac5d02d55c7f1f24992c1ac544069e17eb56875d4dd2bac
Oct 30 06:00:31.062757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.062700    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"]
Oct 30 06:00:31.065089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.065055    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:31.065226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.065139    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:31.065226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.065200    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:31.263840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.263814    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:00:31.363014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.362913    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" event={"ID":"872e7a34-3fe4-4ae5-b29b-26848745a69c","Type":"ContainerStarted","Data":"3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b"}
Oct 30 06:00:31.363014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.362965    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" event={"ID":"872e7a34-3fe4-4ae5-b29b-26848745a69c","Type":"ContainerStarted","Data":"ba8ae0b94c1835c5cac5d02d55c7f1f24992c1ac544069e17eb56875d4dd2bac"}
Oct 30 06:00:31.412237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.412150    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" podStartSLOduration=1.412136305 podStartE2EDuration="1.412136305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:31.410907016 +0000 UTC m=+1746.241944233" watchObservedRunningTime="2026-10-30 06:00:31.412136305 +0000 UTC m=+1746.243173519"
Oct 30 06:00:31.596114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.596073    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:31.729857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:31.729826    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:32.403628 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:32.403594    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:00:32.730804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:32.730714    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 06:00:32.730804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:32.730776    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 06:00:33.483460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:33.483422    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:33.730393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:33.730358    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 06:00:33.730576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:33.730413    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 06:00:34.093030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:34.092990    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:34.538093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:34.538058    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:34.729706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:34.729667    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 06:00:34.729886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:34.729730    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 06:00:35.630968 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:35.630933    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:35.730044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:35.730013    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 06:00:35.730246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:35.730063    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 06:00:36.730292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:36.730258    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 06:00:36.730692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:36.730309    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 06:00:37.621502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:37.621458    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:37.735001 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:37.734971    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:37.735484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:37.735436    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:37.736238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:37.736210    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:00:37.766869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:37.766842    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:37.924405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:37.924367    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:39.564436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:39.564397    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:41.236654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:41.236621    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:41.394127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:41.394092    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:41.595230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:41.595106    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:43.264440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:43.264397    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:43.415415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:43.414646    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:44.975273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:44.975232    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:45.364681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.364588    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"]
Oct 30 06:00:45.365482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365459    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:45.365482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365482    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:45.365650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365515    5250 cpu_manager.go:414] "containerMap: removing container" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:45.365650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365522    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afaa4574-c8d1-4548-a2c7-42924f05c770" containerName="cleanup"
Oct 30 06:00:45.365650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365545    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:45.365650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365552    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:45.365650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365574    5250 cpu_manager.go:414] "containerMap: removing container" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:45.365650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365581    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95aa5335-0ce2-4ae1-a7a1-9c2bc3f81c4c" containerName="cleanup"
Oct 30 06:00:45.365953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365742    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="158297dd-1b28-494d-a338-9305cbc7a461" containerName="cleanup"
Oct 30 06:00:45.365953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365764    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2d60ac-6588-46c3-a44a-3f381e576175" containerName="cleanup"
Oct 30 06:00:45.365953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.365776    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df893bc-ced3-4c63-a549-74f58bcd0423" containerName="cleanup"
Oct 30 06:00:45.371069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.371047    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.390465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.390434    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"]
Oct 30 06:00:45.435350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.435298    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.435560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.435374    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.435560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.435495    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.435693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.435616    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.435734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.435720    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e746240-c132-4275-b437-c7f9e65ae25c-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.435789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.435770    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e746240-c132-4275-b437-c7f9e65ae25c-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.435907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.435884    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgpfk\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-kube-api-access-cgpfk\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.435994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.435979    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.436066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.436029    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.537875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.537839    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.537890    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538032    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538131 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538068    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538124    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538241    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538333    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e746240-c132-4275-b437-c7f9e65ae25c-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538372    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538429    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e746240-c132-4275-b437-c7f9e65ae25c-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538435    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538466    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgpfk\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-kube-api-access-cgpfk\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538653    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.538798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.538666    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.539043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.539021    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e746240-c132-4275-b437-c7f9e65ae25c-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.540670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.540647    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.541117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.541099    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e746240-c132-4275-b437-c7f9e65ae25c-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.561168 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.561136    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.561168 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.561150    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgpfk\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-kube-api-access-cgpfk\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:45.687339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:45.687310    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:46.027127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.027093    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"]
Oct 30 06:00:46.027526 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:00:46.027418    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e746240_c132_4275_b437_c7f9e65ae25c.slice/crio-ec0cb7159f9ee51ff5584eeec8c052f93fff2ff4ac4c91eef9be1ca1ddb34d42 WatchSource:0}: Error finding container ec0cb7159f9ee51ff5584eeec8c052f93fff2ff4ac4c91eef9be1ca1ddb34d42: Status 404 returned error can't find the container with id ec0cb7159f9ee51ff5584eeec8c052f93fff2ff4ac4c91eef9be1ca1ddb34d42
Oct 30 06:00:46.030527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.030495    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:46.030628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.030562    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:46.030628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.030593    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:00:46.534315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.534260    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" event={"ID":"8e746240-c132-4275-b437-c7f9e65ae25c","Type":"ContainerStarted","Data":"389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044"}
Oct 30 06:00:46.534502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.534355    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" event={"ID":"8e746240-c132-4275-b437-c7f9e65ae25c","Type":"ContainerStarted","Data":"ec0cb7159f9ee51ff5584eeec8c052f93fff2ff4ac4c91eef9be1ca1ddb34d42"}
Oct 30 06:00:46.594237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.594202    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:46.604833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.604784    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" podStartSLOduration=1.6047649229999998 podStartE2EDuration="1.604764923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:46.592463019 +0000 UTC m=+1761.423500240" watchObservedRunningTime="2026-10-30 06:00:46.604764923 +0000 UTC m=+1761.435802142"
Oct 30 06:00:46.688723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:46.688690    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:47.690106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:47.690072    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 06:00:47.690618 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:47.690125    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 06:00:48.478782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:48.478744    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:48.687656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:48.687623    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 06:00:48.687850 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:48.687680    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 06:00:49.687703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:49.687666    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 06:00:49.688134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:49.687723    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 06:00:49.882796 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:49.882757    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"]
Oct 30 06:00:49.883005 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:49.882980    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy" containerID="cri-o://3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01" gracePeriod=30
Oct 30 06:00:50.138085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:50.138054    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:50.687627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:50.687597    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 06:00:50.687792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:50.687650    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 06:00:51.688235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:51.688158    5250 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 06:00:51.688690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:51.688253    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 06:00:51.689010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:51.688971    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:51.890286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:51.890254    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:52.693054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:52.693024    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:52.693570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:52.693542    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:52.694242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:52.694224    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:00:53.481788 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:53.481747    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:54.507946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:54.507903    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:00:54.508387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:54.507963    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:00:55.172506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.172391    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:55.252336 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.252286    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 06:00:55.332223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332155    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-istio-token\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-istio-token\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.332417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332304    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51df8972-896d-4d45-9975-6cf732c8f699-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51df8972-896d-4d45-9975-6cf732c8f699-istio-podinfo\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.332417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332371    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-socket\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.332543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332423    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-data\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.332543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332479    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-credential-socket\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.332543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332511    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-envoy\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.332715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332628    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51df8972-896d-4d45-9975-6cf732c8f699-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51df8972-896d-4d45-9975-6cf732c8f699-istiod-ca-cert\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.332715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332693    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-certs\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.332830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332720    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-data" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:00:55.332830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.332789    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-credential-socket" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:00:55.333025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333000    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-kube-api-access-hdw5k\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-kube-api-access-hdw5k\") pod \"51df8972-896d-4d45-9975-6cf732c8f699\" (UID: \"51df8972-896d-4d45-9975-6cf732c8f699\") "
Oct 30 06:00:55.333133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333067    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51df8972-896d-4d45-9975-6cf732c8f699-istiod-ca-cert" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:00:55.333385 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333362    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-socket" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:00:55.333490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333451    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-certs" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:00:55.333966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333945    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.333966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333967    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51df8972-896d-4d45-9975-6cf732c8f699-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.334123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333976    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.334123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333984    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.334123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.333994    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.335226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.335166    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51df8972-896d-4d45-9975-6cf732c8f699-istio-podinfo" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:00:55.335328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.335240    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-istio-token" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:55.335543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.335519    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-envoy" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:00:55.335715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.335696    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-kube-api-access-hdw5k" pod "51df8972-896d-4d45-9975-6cf732c8f699" (UID: "51df8972-896d-4d45-9975-6cf732c8f699"). InnerVolumeSpecName "kube-api-access-hdw5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:00:55.403333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.403293    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:55.435346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.435264    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdw5k\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-kube-api-access-hdw5k\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.435346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.435300    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51df8972-896d-4d45-9975-6cf732c8f699-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.435346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.435313    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51df8972-896d-4d45-9975-6cf732c8f699-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.435346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.435321    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51df8972-896d-4d45-9975-6cf732c8f699-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:00:55.634968 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.634937    5250 generic.go:356] "Generic (PLEG): container finished" podID="51df8972-896d-4d45-9975-6cf732c8f699" containerID="3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01" exitCode=0
Oct 30 06:00:55.635691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.634988    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"
Oct 30 06:00:55.635691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.635001    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" event={"ID":"51df8972-896d-4d45-9975-6cf732c8f699","Type":"ContainerDied","Data":"3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01"}
Oct 30 06:00:55.635691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.635043    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5" event={"ID":"51df8972-896d-4d45-9975-6cf732c8f699","Type":"ContainerDied","Data":"17fadbdf25b61b929c796676d2067b0c7594495d82daa9dbd0d3e9abf500621b"}
Oct 30 06:00:55.635691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.635093    5250 scope.go:122] "RemoveContainer" containerID="3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01"
Oct 30 06:00:55.647207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.647170    5250 scope.go:122] "RemoveContainer" containerID="3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01"
Oct 30 06:00:55.647596 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:00:55.647575    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01\": container with ID starting with 3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01 not found: ID does not exist" containerID="3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01"
Oct 30 06:00:55.647670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.647607    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01"} err="failed to get container status \"3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01\": rpc error: code = NotFound desc = could not find container \"3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01\": container with ID starting with 3c2b55ad617ba91a9c21e32a507926ae089db233585c8e87e1c1e8d092519e01 not found: ID does not exist"
Oct 30 06:00:55.690626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.690552    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"]
Oct 30 06:00:55.701243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:55.701214    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-wwfw5"]
Oct 30 06:00:57.101834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:57.101805    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:57.292371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:57.292320    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51df8972-896d-4d45-9975-6cf732c8f699" path="/var/lib/kubelet/pods/51df8972-896d-4d45-9975-6cf732c8f699/volumes"
Oct 30 06:00:57.820800 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:57.820769    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:57.962921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:57.962884    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:00:58.885363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:00:58.885326    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:00.514893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:00.514714    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:02.344038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:02.343996    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:02.382732 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:01:02.382696    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:01:03.186755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:03.186713    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:04.434014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:04.433972    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:04.587622 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:04.587585    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:05.177660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:05.177627    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:07.253830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:07.253781    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:09.389658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:09.389628    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:12.082984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:12.082923    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:14.190264 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:14.190229    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"]
Oct 30 06:01:14.263446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:14.263411    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:14.501416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:14.501323    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:16.611886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:16.611845    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:18.234346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:18.234310    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:18.529302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:18.529197    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:18.749399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:18.749364    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:19.175398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.175364    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"]
Oct 30 06:01:19.176166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.176147    5250 cpu_manager.go:414] "containerMap: removing container" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy"
Oct 30 06:01:19.176270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.176168    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy"
Oct 30 06:01:19.176445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.176429    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="51df8972-896d-4d45-9975-6cf732c8f699" containerName="istio-proxy"
Oct 30 06:01:19.180322 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.180286    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.196542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.196515    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-02ceed-openshift-default-dockercfg-m7z89\""
Oct 30 06:01:19.213642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.213612    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"]
Oct 30 06:01:19.266930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.266887    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:19.271038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271005    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377ba94-aefe-4012-b1eb-363b2ba740d8-istiod-ca-cert\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.271204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271104    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.271204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271168    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqn4x\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-kube-api-access-zqn4x\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.271329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271221    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-envoy\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.271329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271252    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-token\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.271411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271389    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-credential-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.271449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271430    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-podinfo\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.271485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271464    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-certs\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.271485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.271480    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-data\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.375648 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.375607    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqn4x\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-kube-api-access-zqn4x\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.375955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.375674    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-envoy\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.375955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.375710    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-token\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.376145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.375976    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-credential-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.376145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.376048    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-podinfo\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.376145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.376101    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-certs\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.376145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.376127    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-data\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.376600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.376167    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377ba94-aefe-4012-b1eb-363b2ba740d8-istiod-ca-cert\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.376600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.376323    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.377008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.376718    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.383534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.383502    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-data\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.383980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.383955    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-certs\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.384963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.384789    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377ba94-aefe-4012-b1eb-363b2ba740d8-istiod-ca-cert\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.385650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.385622    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-credential-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.385835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.385780    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-envoy\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.389390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.389364    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-podinfo\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.407720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.401870    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqn4x\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-kube-api-access-zqn4x\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.408041 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.408015    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-token\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.487010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.486693    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:19.505522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.505486    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:19.991138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:19.991041    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:20.003872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:20.003842    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"]
Oct 30 06:01:20.004372 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:20.004349    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode377ba94_aefe_4012_b1eb_363b2ba740d8.slice/crio-553de14e3ba7f6cc76208dc07ac75ba4f2e59941391d519a948054d0a1cecea6 WatchSource:0}: Error finding container 553de14e3ba7f6cc76208dc07ac75ba4f2e59941391d519a948054d0a1cecea6: Status 404 returned error can't find the container with id 553de14e3ba7f6cc76208dc07ac75ba4f2e59941391d519a948054d0a1cecea6
Oct 30 06:01:20.007208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:20.007158    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:01:20.007318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:20.007241    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:01:20.007318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:20.007272    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:01:20.867815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:20.867780    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" event={"ID":"e377ba94-aefe-4012-b1eb-363b2ba740d8","Type":"ContainerStarted","Data":"7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75"}
Oct 30 06:01:20.868380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:20.867834    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" event={"ID":"e377ba94-aefe-4012-b1eb-363b2ba740d8","Type":"ContainerStarted","Data":"553de14e3ba7f6cc76208dc07ac75ba4f2e59941391d519a948054d0a1cecea6"}
Oct 30 06:01:21.043409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:21.043349    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:21.270204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:21.266473    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:21.567775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:21.567698    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:21.573954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:21.573927    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:21.618873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:21.618806    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" podStartSLOduration=2.6187869150000003 podStartE2EDuration="2.618786915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:20.933570398 +0000 UTC m=+1795.764607620" watchObservedRunningTime="2026-10-30 06:01:21.618786915 +0000 UTC m=+1796.449824133"
Oct 30 06:01:21.878983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:21.878900    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:21.880019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:21.879994    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:01:22.878927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:22.878506    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:23.726064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:23.726021    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:24.050713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:24.050605    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:25.477300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:25.477267    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:25.673774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:25.673737    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:25.717246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:25.717212    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:01:25.806267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:25.806154    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:01:26.460787 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:26.460748    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:26.604507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:26.604464    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:26.895548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:26.895503    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:27.064491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:27.064458    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:27.222994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:27.222897    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:27.649714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:27.649675    5250 scope.go:122] "RemoveContainer" containerID="547e9ebf131cda429e8beea6ff89a906134b29cfc0d86b2646e69e6a887b55f8"
Oct 30 06:01:28.757332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:28.757301    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:28.933226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:28.932919    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"]
Oct 30 06:01:28.992200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:28.991495    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:29.006292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:28.999903    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"]
Oct 30 06:01:29.010221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.007407    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"]
Oct 30 06:01:29.015206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.012680    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d917a1\""
Oct 30 06:01:29.031685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.031495    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:29.060985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.059349    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d917a1-dockercfg-rkvdq\""
Oct 30 06:01:29.064302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.063680    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:29.087341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.064843    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1f98e0-9947-4419-9a72-40efa3015dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9\" (UID: \"ba1f98e0-9947-4419-9a72-40efa3015dae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:29.087341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.064908    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce90f27-00c2-46df-b6f9-87a760f71466-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl\" (UID: \"5ce90f27-00c2-46df-b6f9-87a760f71466\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:29.087341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.065168    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhq4\" (UniqueName: \"kubernetes.io/projected/5ce90f27-00c2-46df-b6f9-87a760f71466-kube-api-access-8hhq4\") pod \"payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl\" (UID: \"5ce90f27-00c2-46df-b6f9-87a760f71466\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:29.087341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.080440    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6lk\" (UniqueName: \"kubernetes.io/projected/ba1f98e0-9947-4419-9a72-40efa3015dae-kube-api-access-rv6lk\") pod \"payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9\" (UID: \"ba1f98e0-9947-4419-9a72-40efa3015dae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:29.087341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.087113    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"]
Oct 30 06:01:29.182322 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.182289    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhq4\" (UniqueName: \"kubernetes.io/projected/5ce90f27-00c2-46df-b6f9-87a760f71466-kube-api-access-8hhq4\") pod \"payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl\" (UID: \"5ce90f27-00c2-46df-b6f9-87a760f71466\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:29.182479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.182332    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6lk\" (UniqueName: \"kubernetes.io/projected/ba1f98e0-9947-4419-9a72-40efa3015dae-kube-api-access-rv6lk\") pod \"payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9\" (UID: \"ba1f98e0-9947-4419-9a72-40efa3015dae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:29.182479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.182442    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1f98e0-9947-4419-9a72-40efa3015dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9\" (UID: \"ba1f98e0-9947-4419-9a72-40efa3015dae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:29.182580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.182483    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce90f27-00c2-46df-b6f9-87a760f71466-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl\" (UID: \"5ce90f27-00c2-46df-b6f9-87a760f71466\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:29.183237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.183214    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1f98e0-9947-4419-9a72-40efa3015dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9\" (UID: \"ba1f98e0-9947-4419-9a72-40efa3015dae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:29.183237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.183227    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce90f27-00c2-46df-b6f9-87a760f71466-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl\" (UID: \"5ce90f27-00c2-46df-b6f9-87a760f71466\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:29.206937 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.206902    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhq4\" (UniqueName: \"kubernetes.io/projected/5ce90f27-00c2-46df-b6f9-87a760f71466-kube-api-access-8hhq4\") pod \"payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl\" (UID: \"5ce90f27-00c2-46df-b6f9-87a760f71466\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:29.207230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.207169    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6lk\" (UniqueName: \"kubernetes.io/projected/ba1f98e0-9947-4419-9a72-40efa3015dae-kube-api-access-rv6lk\") pod \"payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9\" (UID: \"ba1f98e0-9947-4419-9a72-40efa3015dae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:29.223007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.222957    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p" podUID="4a4049cd-530d-4fa0-9f01-9671302606bf" containerName="main" containerID="cri-o://b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed" gracePeriod=45
Oct 30 06:01:29.398551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.398516    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:29.425953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.422934    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:29.560252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.559850    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 06:01:29.694326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694294    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-model-cache\") pod \"4a4049cd-530d-4fa0-9f01-9671302606bf\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") "
Oct 30 06:01:29.694510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694372    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4049cd-530d-4fa0-9f01-9671302606bf-kube-api-access-df2dw\" (UniqueName: \"kubernetes.io/projected/4a4049cd-530d-4fa0-9f01-9671302606bf-kube-api-access-df2dw\") pod \"4a4049cd-530d-4fa0-9f01-9671302606bf\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") "
Oct 30 06:01:29.694510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694454    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-dshm\") pod \"4a4049cd-530d-4fa0-9f01-9671302606bf\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") "
Oct 30 06:01:29.694595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694528    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-tmp-dir\") pod \"4a4049cd-530d-4fa0-9f01-9671302606bf\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") "
Oct 30 06:01:29.694595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694567    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-home\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-home\") pod \"4a4049cd-530d-4fa0-9f01-9671302606bf\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") "
Oct 30 06:01:29.694662 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694586    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-model-cache" pod "4a4049cd-530d-4fa0-9f01-9671302606bf" (UID: "4a4049cd-530d-4fa0-9f01-9671302606bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:29.694662 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694649    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4049cd-530d-4fa0-9f01-9671302606bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4049cd-530d-4fa0-9f01-9671302606bf-tls-certs\") pod \"4a4049cd-530d-4fa0-9f01-9671302606bf\" (UID: \"4a4049cd-530d-4fa0-9f01-9671302606bf\") "
Oct 30 06:01:29.694798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694777    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-tmp-dir" pod "4a4049cd-530d-4fa0-9f01-9671302606bf" (UID: "4a4049cd-530d-4fa0-9f01-9671302606bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:29.694911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.694883    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-home" pod "4a4049cd-530d-4fa0-9f01-9671302606bf" (UID: "4a4049cd-530d-4fa0-9f01-9671302606bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:29.695708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.695684    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:29.695774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.695715    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:29.695774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.695730    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:29.696965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.696938    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4049cd-530d-4fa0-9f01-9671302606bf-tls-certs" pod "4a4049cd-530d-4fa0-9f01-9671302606bf" (UID: "4a4049cd-530d-4fa0-9f01-9671302606bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:01:29.697075 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.696969    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-dshm" pod "4a4049cd-530d-4fa0-9f01-9671302606bf" (UID: "4a4049cd-530d-4fa0-9f01-9671302606bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:29.697075 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.697002    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4049cd-530d-4fa0-9f01-9671302606bf-kube-api-access-df2dw" pod "4a4049cd-530d-4fa0-9f01-9671302606bf" (UID: "4a4049cd-530d-4fa0-9f01-9671302606bf"). InnerVolumeSpecName "kube-api-access-df2dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:01:29.764812 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:29.764782    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce90f27_00c2_46df_b6f9_87a760f71466.slice/crio-4ea76657b34b14ad2c73f5193af87e0e409572d1ca9bdeec5b12f0748909cca7 WatchSource:0}: Error finding container 4ea76657b34b14ad2c73f5193af87e0e409572d1ca9bdeec5b12f0748909cca7: Status 404 returned error can't find the container with id 4ea76657b34b14ad2c73f5193af87e0e409572d1ca9bdeec5b12f0748909cca7
Oct 30 06:01:29.765246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.765224    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"]
Oct 30 06:01:29.780749 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.780724    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"]
Oct 30 06:01:29.782410 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:29.782380    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1f98e0_9947_4419_9a72_40efa3015dae.slice/crio-81a78ad28a873bc687fc0355500a762ab5bbe7898f1025930509eb0d57950b7e WatchSource:0}: Error finding container 81a78ad28a873bc687fc0355500a762ab5bbe7898f1025930509eb0d57950b7e: Status 404 returned error can't find the container with id 81a78ad28a873bc687fc0355500a762ab5bbe7898f1025930509eb0d57950b7e
Oct 30 06:01:29.797095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.797071    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4049cd-530d-4fa0-9f01-9671302606bf-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:29.797095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.797096    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df2dw\" (UniqueName: \"kubernetes.io/projected/4a4049cd-530d-4fa0-9f01-9671302606bf-kube-api-access-df2dw\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:29.797317 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.797106    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4049cd-530d-4fa0-9f01-9671302606bf-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:29.998809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.998706    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl" event={"ID":"5ce90f27-00c2-46df-b6f9-87a760f71466","Type":"ContainerStarted","Data":"b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c"}
Oct 30 06:01:29.999086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.999061    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl" event={"ID":"5ce90f27-00c2-46df-b6f9-87a760f71466","Type":"ContainerStarted","Data":"4ea76657b34b14ad2c73f5193af87e0e409572d1ca9bdeec5b12f0748909cca7"}
Oct 30 06:01:29.999274 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:29.999259    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:30.003574 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.003547    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9" event={"ID":"ba1f98e0-9947-4419-9a72-40efa3015dae","Type":"ContainerStarted","Data":"541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2"}
Oct 30 06:01:30.003766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.003591    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9" event={"ID":"ba1f98e0-9947-4419-9a72-40efa3015dae","Type":"ContainerStarted","Data":"81a78ad28a873bc687fc0355500a762ab5bbe7898f1025930509eb0d57950b7e"}
Oct 30 06:01:30.003766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.003618    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:30.005612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.005577    5250 generic.go:356] "Generic (PLEG): container finished" podID="4a4049cd-530d-4fa0-9f01-9671302606bf" containerID="b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed" exitCode=0
Oct 30 06:01:30.005737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.005720    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p" event={"ID":"4a4049cd-530d-4fa0-9f01-9671302606bf","Type":"ContainerDied","Data":"b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed"}
Oct 30 06:01:30.005885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.005753    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p" event={"ID":"4a4049cd-530d-4fa0-9f01-9671302606bf","Type":"ContainerDied","Data":"3f95830ada37244e7e3450d28b41db0395359438b37cc671edba4e3c39fb6f4b"}
Oct 30 06:01:30.005885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.005777    5250 scope.go:122] "RemoveContainer" containerID="b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed"
Oct 30 06:01:30.006016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.005940    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"
Oct 30 06:01:30.044631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.044596    5250 scope.go:122] "RemoveContainer" containerID="b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed"
Oct 30 06:01:30.045872 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:01:30.045844    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed\": container with ID starting with b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed not found: ID does not exist" containerID="b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed"
Oct 30 06:01:30.046045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.046016    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed"} err="failed to get container status \"b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed\": rpc error: code = NotFound desc = could not find container \"b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed\": container with ID starting with b6b1c5b7e8155ee48b6bcd9462a89adfe702bf69426fd8af9327673a1a55aeed not found: ID does not exist"
Oct 30 06:01:30.046329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.046280    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl" podStartSLOduration=2.046253676 podStartE2EDuration="2.046253676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:30.044963216 +0000 UTC m=+1804.876000434" watchObservedRunningTime="2026-10-30 06:01:30.046253676 +0000 UTC m=+1804.877290892"
Oct 30 06:01:30.089232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.089160    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9" podStartSLOduration=2.089143401 podStartE2EDuration="2.089143401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:30.085629588 +0000 UTC m=+1804.916666806" watchObservedRunningTime="2026-10-30 06:01:30.089143401 +0000 UTC m=+1804.920180618"
Oct 30 06:01:30.142475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.128928    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"]
Oct 30 06:01:30.142475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:30.135524    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b3e625/test-model-b3e625-kserve-5f844f6c6f-xlm8p"]
Oct 30 06:01:31.283247 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:31.283216    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4049cd-530d-4fa0-9f01-9671302606bf" path="/var/lib/kubelet/pods/4a4049cd-530d-4fa0-9f01-9671302606bf/volumes"
Oct 30 06:01:32.003551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:32.003518    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:32.377622 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:01:32.377527    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:01:33.627577 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:33.627544    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:34.199994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.199958    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:34.517720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.517639    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:34.726138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.725566    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"]
Oct 30 06:01:34.733248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.733204    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy" containerID="cri-o://d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9" gracePeriod=30
Oct 30 06:01:34.740106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.740078    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"]
Oct 30 06:01:34.741500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.741328    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4a4049cd-530d-4fa0-9f01-9671302606bf" containerName="main"
Oct 30 06:01:34.741500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.741353    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4049cd-530d-4fa0-9f01-9671302606bf" containerName="main"
Oct 30 06:01:34.741752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.741624    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4049cd-530d-4fa0-9f01-9671302606bf" containerName="main"
Oct 30 06:01:34.747699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.747675    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.777994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.777414    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"]
Oct 30 06:01:34.800907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.800876    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-38cfad-openshift-default-dockercfg-28vsq\""
Oct 30 06:01:34.828094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.828056    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885392    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-certs\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885473    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-envoy\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885525    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783f198-7706-41b2-8cec-b91b27576482-istiod-ca-cert\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885619    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-istio-token\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885655    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783f198-7706-41b2-8cec-b91b27576482-istio-podinfo\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885819    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hzm\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-kube-api-access-c2hzm\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885879    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885909    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-data\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.887270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.885938    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-credential-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.989682 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.989630    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzm\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-kube-api-access-c2hzm\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.990042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.990016    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.990238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.990221    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-data\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.990389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.990373    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-credential-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.990567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.990539    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-certs\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.990722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.990706    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-envoy\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.992998 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.992978    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783f198-7706-41b2-8cec-b91b27576482-istiod-ca-cert\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.994171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.994152    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-credential-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.995531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.995514    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783f198-7706-41b2-8cec-b91b27576482-istiod-ca-cert\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:34.995867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.995846    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.005201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.997386    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-istio-token\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.005201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.999319    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783f198-7706-41b2-8cec-b91b27576482-istio-podinfo\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.005201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:34.999662    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-certs\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.005201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.000146    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-data\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.005201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.001453    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-envoy\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.008234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.008213    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783f198-7706-41b2-8cec-b91b27576482-istio-podinfo\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.017287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.017263    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hzm\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-kube-api-access-c2hzm\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.020146 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.019875    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-istio-token\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.083331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.083240    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:35.444440 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:35.444410    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb783f198_7706_41b2_8cec_b91b27576482.slice/crio-e4acd4e127bf035bde7f194ccb0d4697eee752687f4c877ee9b22ebecc596589 WatchSource:0}: Error finding container e4acd4e127bf035bde7f194ccb0d4697eee752687f4c877ee9b22ebecc596589: Status 404 returned error can't find the container with id e4acd4e127bf035bde7f194ccb0d4697eee752687f4c877ee9b22ebecc596589
Oct 30 06:01:35.446132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.446103    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"]
Oct 30 06:01:35.447923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.447888    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:01:35.448007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.447954    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:01:35.448007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.447994    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:01:35.837502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:35.837387    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:36.077374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:36.077337    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" event={"ID":"b783f198-7706-41b2-8cec-b91b27576482","Type":"ContainerStarted","Data":"d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34"}
Oct 30 06:01:36.077565 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:36.077400    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" event={"ID":"b783f198-7706-41b2-8cec-b91b27576482","Type":"ContainerStarted","Data":"e4acd4e127bf035bde7f194ccb0d4697eee752687f4c877ee9b22ebecc596589"}
Oct 30 06:01:36.083746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:36.083717    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:36.090134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:36.090080    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:36.128278 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:36.128222    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" podStartSLOduration=2.1282020409999998 podStartE2EDuration="2.128202041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:36.123721136 +0000 UTC m=+1810.954758354" watchObservedRunningTime="2026-10-30 06:01:36.128202041 +0000 UTC m=+1810.959239259"
Oct 30 06:01:36.328113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:36.328075    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:37.085084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:37.085058    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:37.086681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:37.086661    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:01:38.798596 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:38.798564    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:39.054447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:39.054367    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:39.209809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:39.209780    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:40.023050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.022992    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 06:01:40.104890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.104854    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-credential-socket\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.105097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.104916    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-podinfo\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.105097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.104973    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-token\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-token\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.105097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105048    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-certs\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.105097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105091    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-envoy\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.105291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105214    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-credential-socket" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:40.105336 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105308    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b52e50f-e4ab-482b-925a-2dfba4994c19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52e50f-e4ab-482b-925a-2dfba4994c19-istiod-ca-cert\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.105398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105378    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-data\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.105593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105575    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-kube-api-access-jkz7v\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-kube-api-access-jkz7v\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.105724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105698    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b52e50f-e4ab-482b-925a-2dfba4994c19-istiod-ca-cert" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:01:40.105834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105816    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-socket\") pod \"6b52e50f-e4ab-482b-925a-2dfba4994c19\" (UID: \"6b52e50f-e4ab-482b-925a-2dfba4994c19\") "
Oct 30 06:01:40.106329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.105882    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-data" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:40.106329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.106053    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-socket" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:40.106580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.106557    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-certs" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:40.107101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.107078    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52e50f-e4ab-482b-925a-2dfba4994c19-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.107285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.107270    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.107402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.107389    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.107492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.107480    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.107586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.107574    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.110118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.110080    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-podinfo" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:01:40.110453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.110429    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-envoy" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:01:40.110688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.110574    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-kube-api-access-jkz7v" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "kube-api-access-jkz7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:01:40.110688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.110666    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-token" pod "6b52e50f-e4ab-482b-925a-2dfba4994c19" (UID: "6b52e50f-e4ab-482b-925a-2dfba4994c19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:01:40.133656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.133620    5250 generic.go:356] "Generic (PLEG): container finished" podID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerID="d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9" exitCode=0
Oct 30 06:01:40.133826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.133710    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"
Oct 30 06:01:40.133950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.133905    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" event={"ID":"6b52e50f-e4ab-482b-925a-2dfba4994c19","Type":"ContainerDied","Data":"d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9"}
Oct 30 06:01:40.134011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.133960    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" event={"ID":"6b52e50f-e4ab-482b-925a-2dfba4994c19","Type":"ContainerDied","Data":"0f793bf52db8bbc82f388662f74a6b00dce0a1106848f5f8b1ca83dfcac1c104"}
Oct 30 06:01:40.134011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.133991    5250 scope.go:122] "RemoveContainer" containerID="d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9"
Oct 30 06:01:40.154138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.154114    5250 scope.go:122] "RemoveContainer" containerID="d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9"
Oct 30 06:01:40.154446 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:01:40.154423    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9\": container with ID starting with d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9 not found: ID does not exist" containerID="d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9"
Oct 30 06:01:40.154539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.154458    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9"} err="failed to get container status \"d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9\": rpc error: code = NotFound desc = could not find container \"d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9\": container with ID starting with d67d7be16bf1a612ab8125584cbbfd2027f3418449817f575507e51045d410e9 not found: ID does not exist"
Oct 30 06:01:40.189418 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.189393    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"]
Oct 30 06:01:40.196481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.196454    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2"]
Oct 30 06:01:40.208965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.208933    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.208965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.208963    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.208965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.208973    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52e50f-e4ab-482b-925a-2dfba4994c19-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.209283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.208982    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkz7v\" (UniqueName: \"kubernetes.io/projected/6b52e50f-e4ab-482b-925a-2dfba4994c19-kube-api-access-jkz7v\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:01:40.768960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.768929    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:40.962490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.962460    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:40.978783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.978758    5250 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:01:40.978922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:40.978819    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-hlgp2" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:01:41.017694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.017664    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:01:41.017838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.017739    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:01:41.045840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.045756    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"]
Oct 30 06:01:41.046933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.046905    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy"
Oct 30 06:01:41.046933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.046932    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy"
Oct 30 06:01:41.047104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.047093    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" containerName="istio-proxy"
Oct 30 06:01:41.052525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.052503    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"]
Oct 30 06:01:41.052647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.052628    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:41.057010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.056989    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:41.118983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.118955    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"]
Oct 30 06:01:41.135291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.135257    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f1bc22-d6ba-479f-9005-a6a90d51e882-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz\" (UID: \"54f1bc22-d6ba-479f-9005-a6a90d51e882\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:41.135453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.135434    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a153bad2-aa94-4c96-b259-e487efedb6d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh\" (UID: \"a153bad2-aa94-4c96-b259-e487efedb6d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:41.135495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.135476    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zln\" (UniqueName: \"kubernetes.io/projected/54f1bc22-d6ba-479f-9005-a6a90d51e882-kube-api-access-m2zln\") pod \"payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz\" (UID: \"54f1bc22-d6ba-479f-9005-a6a90d51e882\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:41.135535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.135521    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmh7\" (UniqueName: \"kubernetes.io/projected/a153bad2-aa94-4c96-b259-e487efedb6d0-kube-api-access-8nmh7\") pod \"payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh\" (UID: \"a153bad2-aa94-4c96-b259-e487efedb6d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:41.170702 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.151290    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-38cfad-dockercfg-frps2\""
Oct 30 06:01:41.179007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.178970    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-38cfad\""
Oct 30 06:01:41.189367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.189338    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"]
Oct 30 06:01:41.391229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.361864    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f1bc22-d6ba-479f-9005-a6a90d51e882-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz\" (UID: \"54f1bc22-d6ba-479f-9005-a6a90d51e882\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:41.391229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.362196    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a153bad2-aa94-4c96-b259-e487efedb6d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh\" (UID: \"a153bad2-aa94-4c96-b259-e487efedb6d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:41.391229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.362244    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zln\" (UniqueName: \"kubernetes.io/projected/54f1bc22-d6ba-479f-9005-a6a90d51e882-kube-api-access-m2zln\") pod \"payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz\" (UID: \"54f1bc22-d6ba-479f-9005-a6a90d51e882\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:41.391229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.362317    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmh7\" (UniqueName: \"kubernetes.io/projected/a153bad2-aa94-4c96-b259-e487efedb6d0-kube-api-access-8nmh7\") pod \"payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh\" (UID: \"a153bad2-aa94-4c96-b259-e487efedb6d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:41.391229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.364801    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f1bc22-d6ba-479f-9005-a6a90d51e882-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz\" (UID: \"54f1bc22-d6ba-479f-9005-a6a90d51e882\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:41.391229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.366009    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a153bad2-aa94-4c96-b259-e487efedb6d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh\" (UID: \"a153bad2-aa94-4c96-b259-e487efedb6d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:41.400326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.399729    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmh7\" (UniqueName: \"kubernetes.io/projected/a153bad2-aa94-4c96-b259-e487efedb6d0-kube-api-access-8nmh7\") pod \"payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh\" (UID: \"a153bad2-aa94-4c96-b259-e487efedb6d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:41.405314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.405278    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b52e50f-e4ab-482b-925a-2dfba4994c19" path="/var/lib/kubelet/pods/6b52e50f-e4ab-482b-925a-2dfba4994c19/volumes"
Oct 30 06:01:41.413451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.411073    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zln\" (UniqueName: \"kubernetes.io/projected/54f1bc22-d6ba-479f-9005-a6a90d51e882-kube-api-access-m2zln\") pod \"payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz\" (UID: \"54f1bc22-d6ba-479f-9005-a6a90d51e882\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:41.440061 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.440027    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:41.605413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.605380    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:41.669544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.669450    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:41.676123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:41.676091    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:42.043704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.043679    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"]
Oct 30 06:01:42.044964 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:42.044934    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f1bc22_d6ba_479f_9005_a6a90d51e882.slice/crio-9c3aca6a79cd1f4ccc47d7fdc2920d5612702260e8b617a50fee6350f533dc47 WatchSource:0}: Error finding container 9c3aca6a79cd1f4ccc47d7fdc2920d5612702260e8b617a50fee6350f533dc47: Status 404 returned error can't find the container with id 9c3aca6a79cd1f4ccc47d7fdc2920d5612702260e8b617a50fee6350f533dc47
Oct 30 06:01:42.060894 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:42.060868    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda153bad2_aa94_4c96_b259_e487efedb6d0.slice/crio-cb58bd9a61f093d7b999b5200be00b7c809af25b968ee3dccceec8532a90b67f WatchSource:0}: Error finding container cb58bd9a61f093d7b999b5200be00b7c809af25b968ee3dccceec8532a90b67f: Status 404 returned error can't find the container with id cb58bd9a61f093d7b999b5200be00b7c809af25b968ee3dccceec8532a90b67f
Oct 30 06:01:42.063750 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.063719    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"]
Oct 30 06:01:42.170479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.170439    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz" event={"ID":"54f1bc22-d6ba-479f-9005-a6a90d51e882","Type":"ContainerStarted","Data":"9c3aca6a79cd1f4ccc47d7fdc2920d5612702260e8b617a50fee6350f533dc47"}
Oct 30 06:01:42.172528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.172499    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh" event={"ID":"a153bad2-aa94-4c96-b259-e487efedb6d0","Type":"ContainerStarted","Data":"021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4"}
Oct 30 06:01:42.172652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.172572    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:42.172652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.172598    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh" event={"ID":"a153bad2-aa94-4c96-b259-e487efedb6d0","Type":"ContainerStarted","Data":"cb58bd9a61f093d7b999b5200be00b7c809af25b968ee3dccceec8532a90b67f"}
Oct 30 06:01:42.214956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.214901    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh" podStartSLOduration=2.214883513 podStartE2EDuration="2.214883513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:42.211376893 +0000 UTC m=+1817.042414112" watchObservedRunningTime="2026-10-30 06:01:42.214883513 +0000 UTC m=+1817.045920733"
Oct 30 06:01:42.420818 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.420784    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:42.943655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:42.943628    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:43.182338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:43.182304    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz" event={"ID":"54f1bc22-d6ba-479f-9005-a6a90d51e882","Type":"ContainerStarted","Data":"790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247"}
Oct 30 06:01:43.182809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:43.182389    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:43.240401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:43.240278    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz" podStartSLOduration=3.240260419 podStartE2EDuration="3.240260419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:43.236055974 +0000 UTC m=+1818.067093193" watchObservedRunningTime="2026-10-30 06:01:43.240260419 +0000 UTC m=+1818.071297636"
Oct 30 06:01:43.466891 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:43.466854    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:43.608303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:43.607475    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:44.164516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:44.164474    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:45.314078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:45.314037    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:46.159456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:46.159403    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:47.516473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:47.516437    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:47.713656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:47.713624    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:48.330617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:48.330578    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:49.160011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.159970    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:49.321866 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.321828    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"]
Oct 30 06:01:49.326489 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.326466    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:49.330383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.330359    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:49.349963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.349930    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"]
Oct 30 06:01:49.355643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.355624    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7dfcb7\""
Oct 30 06:01:49.356944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.356547    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7dfcb7-dockercfg-7pc84\""
Oct 30 06:01:49.459348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.442114    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afab366-9076-4bd4-8fd5-220ca11b1f80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk\" (UID: \"6afab366-9076-4bd4-8fd5-220ca11b1f80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:49.460781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.460741    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnnv\" (UniqueName: \"kubernetes.io/projected/6afab366-9076-4bd4-8fd5-220ca11b1f80-kube-api-access-rpnnv\") pod \"payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk\" (UID: \"6afab366-9076-4bd4-8fd5-220ca11b1f80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:49.563037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.562978    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afab366-9076-4bd4-8fd5-220ca11b1f80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk\" (UID: \"6afab366-9076-4bd4-8fd5-220ca11b1f80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:49.565811 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.565784    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnnv\" (UniqueName: \"kubernetes.io/projected/6afab366-9076-4bd4-8fd5-220ca11b1f80-kube-api-access-rpnnv\") pod \"payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk\" (UID: \"6afab366-9076-4bd4-8fd5-220ca11b1f80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:49.566351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.565612    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afab366-9076-4bd4-8fd5-220ca11b1f80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk\" (UID: \"6afab366-9076-4bd4-8fd5-220ca11b1f80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:49.572517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.572493    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"]
Oct 30 06:01:49.578217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.578197    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"]
Oct 30 06:01:49.578348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.578325    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:49.590786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.590761    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnnv\" (UniqueName: \"kubernetes.io/projected/6afab366-9076-4bd4-8fd5-220ca11b1f80-kube-api-access-rpnnv\") pod \"payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk\" (UID: \"6afab366-9076-4bd4-8fd5-220ca11b1f80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:49.643468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.643433    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:49.669154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.669119    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmwr\" (UniqueName: \"kubernetes.io/projected/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-kube-api-access-sxmwr\") pod \"payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr\" (UID: \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:49.669382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.669364    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr\" (UID: \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:49.772256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.771751    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr\" (UID: \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:49.772256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.771909    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmwr\" (UniqueName: \"kubernetes.io/projected/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-kube-api-access-sxmwr\") pod \"payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr\" (UID: \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:49.778578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.773262    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr\" (UID: \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:49.822148 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.822114    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmwr\" (UniqueName: \"kubernetes.io/projected/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-kube-api-access-sxmwr\") pod \"payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr\" (UID: \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:49.919993 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:49.919964    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:50.054658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:50.054577    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"]
Oct 30 06:01:50.060341 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:50.060306    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afab366_9076_4bd4_8fd5_220ca11b1f80.slice/crio-bd6edb820b6351030839e55a92d049bc2dafa6d7e3a622b9a909658e5f192383 WatchSource:0}: Error finding container bd6edb820b6351030839e55a92d049bc2dafa6d7e3a622b9a909658e5f192383: Status 404 returned error can't find the container with id bd6edb820b6351030839e55a92d049bc2dafa6d7e3a622b9a909658e5f192383
Oct 30 06:01:50.238453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:50.238424    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"]
Oct 30 06:01:50.239359 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:50.239332    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfdf965_fc79_496e_b6b6_a0c6e1807df8.slice/crio-f059363d8f67e72415fc9abb7e8960d61eff2621170f0360253f0c63867bdad3 WatchSource:0}: Error finding container f059363d8f67e72415fc9abb7e8960d61eff2621170f0360253f0c63867bdad3: Status 404 returned error can't find the container with id f059363d8f67e72415fc9abb7e8960d61eff2621170f0360253f0c63867bdad3
Oct 30 06:01:50.249028 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:50.249000    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr" event={"ID":"0bfdf965-fc79-496e-b6b6-a0c6e1807df8","Type":"ContainerStarted","Data":"f059363d8f67e72415fc9abb7e8960d61eff2621170f0360253f0c63867bdad3"}
Oct 30 06:01:50.250845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:50.250818    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk" event={"ID":"6afab366-9076-4bd4-8fd5-220ca11b1f80","Type":"ContainerStarted","Data":"97c0fb0b8d58de3af39842554752b74665970a17d4eb0f1a9159b201973b82a4"}
Oct 30 06:01:50.250977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:50.250868    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk" event={"ID":"6afab366-9076-4bd4-8fd5-220ca11b1f80","Type":"ContainerStarted","Data":"bd6edb820b6351030839e55a92d049bc2dafa6d7e3a622b9a909658e5f192383"}
Oct 30 06:01:50.250977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:50.250945    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:01:50.295341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:50.295290    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk" podStartSLOduration=1.29527402 podStartE2EDuration="1.29527402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:50.293636244 +0000 UTC m=+1825.124673460" watchObservedRunningTime="2026-10-30 06:01:50.29527402 +0000 UTC m=+1825.126311237"
Oct 30 06:01:50.696378 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:50.696340    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:51.200367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.200331    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"]
Oct 30 06:01:51.206016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.205983    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.225072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.225043    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"]
Oct 30 06:01:51.231262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.231234    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0382c2\"/\"openshift-service-ca.crt\""
Oct 30 06:01:51.231438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.231256    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0382c2\"/\"kube-root-ca.crt\""
Oct 30 06:01:51.231502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.231302    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0382c2\"/\"default-dockercfg-rgt2c\""
Oct 30 06:01:51.236258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.236236    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0382c2\"/\"test-model-0382c2-kserve-self-signed-certs\""
Oct 30 06:01:51.261718 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.261687    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr" event={"ID":"0bfdf965-fc79-496e-b6b6-a0c6e1807df8","Type":"ContainerStarted","Data":"a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107"}
Oct 30 06:01:51.262154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.261744    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:01:51.288497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.288461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:51.294390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.294358    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9rb\" (UniqueName: \"kubernetes.io/projected/0e7991cb-1a59-46de-8fa8-c228c7e4033d-kube-api-access-mx9rb\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.294532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.294462    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-dshm\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.294691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.294656    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-home\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.294785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.294752    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tls-certs\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.295008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.294978    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-model-cache\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.295130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.295110    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tmp-dir\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.307663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.307614    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr" podStartSLOduration=2.30760094 podStartE2EDuration="2.30760094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:51.303118296 +0000 UTC m=+1826.134155513" watchObservedRunningTime="2026-10-30 06:01:51.30760094 +0000 UTC m=+1826.138638156"
Oct 30 06:01:51.396960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.396916    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-model-cache\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.397156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.397005    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tmp-dir\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.397156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.397123    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9rb\" (UniqueName: \"kubernetes.io/projected/0e7991cb-1a59-46de-8fa8-c228c7e4033d-kube-api-access-mx9rb\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.397311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.397222    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-dshm\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.397311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.397250    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-home\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.397311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.397278    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tls-certs\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.397472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.397388    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-model-cache\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.397472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.397400    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tmp-dir\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.397651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.397633    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-home\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.399758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.399730    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-dshm\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.399982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.399965    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tls-certs\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.419998 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.419972    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9rb\" (UniqueName: \"kubernetes.io/projected/0e7991cb-1a59-46de-8fa8-c228c7e4033d-kube-api-access-mx9rb\") pod \"test-model-0382c2-kserve-56b886478c-lxzrl\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") " pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.549730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.549617    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:51.987883 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:51.987831    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"]
Oct 30 06:01:51.988482 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:01:51.988454    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e7991cb_1a59_46de_8fa8_c228c7e4033d.slice/crio-3ea0c905337e6a517d920a114fa58d01aac206fd11238d5ccbd754bc6e4715ed WatchSource:0}: Error finding container 3ea0c905337e6a517d920a114fa58d01aac206fd11238d5ccbd754bc6e4715ed: Status 404 returned error can't find the container with id 3ea0c905337e6a517d920a114fa58d01aac206fd11238d5ccbd754bc6e4715ed
Oct 30 06:01:52.271760 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:52.271671    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl" event={"ID":"0e7991cb-1a59-46de-8fa8-c228c7e4033d","Type":"ContainerStarted","Data":"607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91"}
Oct 30 06:01:52.271760 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:52.271714    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl" event={"ID":"0e7991cb-1a59-46de-8fa8-c228c7e4033d","Type":"ContainerStarted","Data":"3ea0c905337e6a517d920a114fa58d01aac206fd11238d5ccbd754bc6e4715ed"}
Oct 30 06:01:52.315679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:52.315629    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl" podStartSLOduration=1.315615093 podStartE2EDuration="1.315615093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:52.310954817 +0000 UTC m=+1827.141992035" watchObservedRunningTime="2026-10-30 06:01:52.315615093 +0000 UTC m=+1827.146652309"
Oct 30 06:01:52.550733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:52.550647    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:01:52.799322 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:52.799285    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:53.185837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:53.185802    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:01:54.198231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:54.198201    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:01:54.633658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:54.633628    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:56.092499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:56.092454    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:56.244031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:56.243993    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:56.444699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:56.444654    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:01:58.858213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:01:58.858161    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:00.810214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:00.810148    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:01.265306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:01.265272    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:02:01.540115 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:01.540036    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:01.550738 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:01.550708    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:02:01.566774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:01.566741    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:02:02.295250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:02.294585    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:02:02.411668 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:02:02.411627    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:02:02.426371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:02.426343    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:02:03.465079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:03.465021    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:05.355047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:05.355006    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:07.097494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:07.097449    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:09.030094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:09.030044    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:10.661478 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:10.661439    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:12.280397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:12.280364    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:13.348651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:13.348617    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:14.146329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:14.146298    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:14.291015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:14.290947    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:15.235130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:15.235100    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:15.665073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:15.664995    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:16.265483 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:16.265447    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:16.511679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:16.507931    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:17.220868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:17.220830    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:17.418708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:17.418668    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:19.278974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:19.278935    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:19.425328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:19.425280    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:19.625946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:19.625861    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:21.070244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:21.065660    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:21.608135 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:21.608099    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:23.236923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.236883    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"]
Oct 30 06:02:23.242061 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.242040    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.267217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.267156    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d917a1-serving-cert\""
Oct 30 06:02:23.288342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.288312    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"]
Oct 30 06:02:23.348290 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.348246    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pzf\" (UniqueName: \"kubernetes.io/projected/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-kube-api-access-j5pzf\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.348468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.348299    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-maas-api-tls\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.348521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.348470    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.348663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.348646    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-metrics-certs\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.367529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.367495    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:23.451027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.450960    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-metrics-certs\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.451516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.451486    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pzf\" (UniqueName: \"kubernetes.io/projected/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-kube-api-access-j5pzf\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.451656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.451533    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-maas-api-tls\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.451656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.451649    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.457236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.455742    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.457236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.456847    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-maas-api-tls\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.460790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.460767    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-metrics-certs\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.477074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.477041    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pzf\" (UniqueName: \"kubernetes.io/projected/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-kube-api-access-j5pzf\") pod \"maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.511497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.511411    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:23.557341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.557310    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:23.668362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.668329    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:23.827622 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.827542    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:23.875084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.875060    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"]
Oct 30 06:02:23.877210 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:02:23.877160    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43f3c3a_3d30_4744_aaed_8d39c60f71e6.slice/crio-b5caeabc5bce41aa1f1b1b736ab56de5dda27a5236500aeecc7119a6c84b1ea4 WatchSource:0}: Error finding container b5caeabc5bce41aa1f1b1b736ab56de5dda27a5236500aeecc7119a6c84b1ea4: Status 404 returned error can't find the container with id b5caeabc5bce41aa1f1b1b736ab56de5dda27a5236500aeecc7119a6c84b1ea4
Oct 30 06:02:23.878831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:23.878812    5250 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 06:02:24.070685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:24.070649    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:24.666643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:24.666606    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45" event={"ID":"e43f3c3a-3d30-4744-aaed-8d39c60f71e6","Type":"ContainerStarted","Data":"1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6"}
Oct 30 06:02:24.667044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:24.666667    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45" event={"ID":"e43f3c3a-3d30-4744-aaed-8d39c60f71e6","Type":"ContainerStarted","Data":"b5caeabc5bce41aa1f1b1b736ab56de5dda27a5236500aeecc7119a6c84b1ea4"}
Oct 30 06:02:24.667044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:24.666695    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:24.708002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:24.707906    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45" podStartSLOduration=1.148517615 podStartE2EDuration="1.707891803s" totalImagesPullingTime="559.374188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:24.706635919 +0000 UTC m=+1859.537673137" watchObservedRunningTime="2026-10-30 06:02:24.707891803 +0000 UTC m=+1859.538929019"
Oct 30 06:02:25.615915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:25.613670    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:27.504739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:27.504659    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:29.398012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:29.397945    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:30.695028 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:30.694994    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:02:31.157540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:31.157498    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:31.301452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:31.301373    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:31.552098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:31.552011    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:31.703451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:31.703420    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:31.853587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:31.853496    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:32.433243 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:02:32.426389    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:02:33.284821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:33.284783    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:34.579964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:34.579932    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:34.988421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:34.988386    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:36.118726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:36.118689    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"]
Oct 30 06:02:36.237949 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:36.237915    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:36.575447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:36.575362    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:36.600030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:36.599977    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"]
Oct 30 06:02:36.618776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:36.618737    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8" podUID="2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" containerName="maas-api" containerID="cri-o://4a9a5ac6095ca5774ff97d7d49e920d733fb46cf55150490611a3fc6f1ae5dd3" gracePeriod=30
Oct 30 06:02:36.727401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:36.727366    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:36.785792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:36.785746    5250 generic.go:356] "Generic (PLEG): container finished" podID="2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" containerID="4a9a5ac6095ca5774ff97d7d49e920d733fb46cf55150490611a3fc6f1ae5dd3" exitCode=0
Oct 30 06:02:36.786015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:36.785953    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8" event={"ID":"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5","Type":"ContainerDied","Data":"4a9a5ac6095ca5774ff97d7d49e920d733fb46cf55150490611a3fc6f1ae5dd3"}
Oct 30 06:02:37.430087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.430060    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 06:02:37.559212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.556865    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-maas-api-tls\") pod \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") "
Oct 30 06:02:37.559212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.556937    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-kube-api-access-4qbxr\" (UniqueName: \"kubernetes.io/projected/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-kube-api-access-4qbxr\") pod \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") "
Oct 30 06:02:37.559212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.557010    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-metrics-certs\") pod \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") "
Oct 30 06:02:37.559212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.557265    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-openshift-service-ca\") pod \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\" (UID: \"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5\") "
Oct 30 06:02:37.559212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.558435    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-openshift-service-ca" pod "2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" (UID: "2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:02:37.579270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.571355    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-maas-api-tls" pod "2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" (UID: "2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:02:37.579270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.571495    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-kube-api-access-4qbxr" pod "2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" (UID: "2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5"). InnerVolumeSpecName "kube-api-access-4qbxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:02:37.625394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.622807    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-metrics-certs" pod "2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" (UID: "2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:02:37.684253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.669714    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:02:37.684253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.669758    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:02:37.684253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.669776    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qbxr\" (UniqueName: \"kubernetes.io/projected/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-kube-api-access-4qbxr\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:02:37.684253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.669790    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:02:37.796656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.796612    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8" event={"ID":"2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5","Type":"ContainerDied","Data":"f0d8f300f14a37bddb0458c42294a3d782168ef5ab7b39ef18aa83245a6841c8"}
Oct 30 06:02:37.796656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.796643    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"
Oct 30 06:02:37.796887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.796687    5250 scope.go:122] "RemoveContainer" containerID="4a9a5ac6095ca5774ff97d7d49e920d733fb46cf55150490611a3fc6f1ae5dd3"
Oct 30 06:02:37.844173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.844141    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"]
Oct 30 06:02:37.855727 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.855702    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8"]
Oct 30 06:02:37.923494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:37.923461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:39.283530 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:39.283499    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" path="/var/lib/kubelet/pods/2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5/volumes"
Oct 30 06:02:39.753574 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:39.753538    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:39.907687 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:39.907561    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:40.368245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:40.368198    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39147c-865f885f6f-7wxx8" podUID="2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.128:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:02:41.127232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:41.123261    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk" podUID="c8a35609-ba89-4d0b-9962-d464c2229e3b" containerName="payload-processing" containerID="cri-o://cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7" gracePeriod=25
Oct 30 06:02:41.741872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:41.741841    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:43.392358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:43.392322    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:45.135696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:45.120158    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:46.527154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:46.527121    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk" podUID="c8a35609-ba89-4d0b-9962-d464c2229e3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:02:47.159603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:47.159572    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:49.080791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:49.080754    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:49.680739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:49.680701    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:49.993301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:49.989902    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:52.672560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:52.672517    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:54.677828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:54.677553    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:56.527249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:56.527107    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk" podUID="c8a35609-ba89-4d0b-9962-d464c2229e3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:02:56.828556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:56.828443    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:02:58.989487 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:02:58.989454    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:00.059878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:00.059839    5250 generic.go:356] "Generic (PLEG): container finished" podID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerID="7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75" exitCode=0
Oct 30 06:03:00.060306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:00.059898    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" event={"ID":"e377ba94-aefe-4012-b1eb-363b2ba740d8","Type":"ContainerDied","Data":"7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75"}
Oct 30 06:03:00.060462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:00.060432    5250 scope.go:122] "RemoveContainer" containerID="7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75"
Oct 30 06:03:00.063167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:00.062913    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:03:00.063167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:00.063030    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:03:00.063167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:00.063075    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:03:01.074466 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.074431    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" event={"ID":"e377ba94-aefe-4012-b1eb-363b2ba740d8","Type":"ContainerStarted","Data":"f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e"}
Oct 30 06:03:01.099865 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.099813    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:01.391172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.391141    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 06:03:01.524464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.524434    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a35609-ba89-4d0b-9962-d464c2229e3b-kube-api-access-bv5zf\" (UniqueName: \"kubernetes.io/projected/c8a35609-ba89-4d0b-9962-d464c2229e3b-kube-api-access-bv5zf\") pod \"c8a35609-ba89-4d0b-9962-d464c2229e3b\" (UID: \"c8a35609-ba89-4d0b-9962-d464c2229e3b\") "
Oct 30 06:03:01.524629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.524567    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a35609-ba89-4d0b-9962-d464c2229e3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a35609-ba89-4d0b-9962-d464c2229e3b-plugins-config-volume\") pod \"c8a35609-ba89-4d0b-9962-d464c2229e3b\" (UID: \"c8a35609-ba89-4d0b-9962-d464c2229e3b\") "
Oct 30 06:03:01.524997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.524970    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a35609-ba89-4d0b-9962-d464c2229e3b-plugins-config-volume" pod "c8a35609-ba89-4d0b-9962-d464c2229e3b" (UID: "c8a35609-ba89-4d0b-9962-d464c2229e3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:03:01.525212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.525194    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a35609-ba89-4d0b-9962-d464c2229e3b-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:03:01.526732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.526704    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a35609-ba89-4d0b-9962-d464c2229e3b-kube-api-access-bv5zf" pod "c8a35609-ba89-4d0b-9962-d464c2229e3b" (UID: "c8a35609-ba89-4d0b-9962-d464c2229e3b"). InnerVolumeSpecName "kube-api-access-bv5zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:03:01.577262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.573352    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:03:01.577262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.573848    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 06:03:01.577262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.573925    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 06:03:01.626961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:01.626914    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv5zf\" (UniqueName: \"kubernetes.io/projected/c8a35609-ba89-4d0b-9962-d464c2229e3b-kube-api-access-bv5zf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:03:02.090550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.090514    5250 generic.go:356] "Generic (PLEG): container finished" podID="c8a35609-ba89-4d0b-9962-d464c2229e3b" containerID="cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7" exitCode=0
Oct 30 06:03:02.090999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.090564    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"
Oct 30 06:03:02.090999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.090607    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk" event={"ID":"c8a35609-ba89-4d0b-9962-d464c2229e3b","Type":"ContainerDied","Data":"cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7"}
Oct 30 06:03:02.090999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.090669    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk" event={"ID":"c8a35609-ba89-4d0b-9962-d464c2229e3b","Type":"ContainerDied","Data":"edbf9a5c6e16c813a46c20a45e15df60aeaba237547670c1be9d8894f6ba4522"}
Oct 30 06:03:02.090999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.090693    5250 scope.go:122] "RemoveContainer" containerID="cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7"
Oct 30 06:03:02.102425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.102403    5250 scope.go:122] "RemoveContainer" containerID="cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7"
Oct 30 06:03:02.102694 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:03:02.102675    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7\": container with ID starting with cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7 not found: ID does not exist" containerID="cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7"
Oct 30 06:03:02.102759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.102702    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7"} err="failed to get container status \"cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7\": rpc error: code = NotFound desc = could not find container \"cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7\": container with ID starting with cbe37f761086b8bf89e8b22b0f53248ff8122ddf2ec5da354a190feb9f21e1c7 not found: ID does not exist"
Oct 30 06:03:02.152089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.152049    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"]
Oct 30 06:03:02.183552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.183513    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39147c-5dc5c6bb7b-4h7dk"]
Oct 30 06:03:02.399821 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:03:02.399787    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:03:02.575023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.574563    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 06:03:02.575023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.574632    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 06:03:02.933901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:02.933872    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:03.289558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:03.289471    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a35609-ba89-4d0b-9962-d464c2229e3b" path="/var/lib/kubelet/pods/c8a35609-ba89-4d0b-9962-d464c2229e3b/volumes"
Oct 30 06:03:03.581395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:03.581292    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 06:03:03.581395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:03.581359    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 06:03:04.577205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:04.576632    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 06:03:04.577205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:04.576704    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 06:03:04.891335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:04.891280    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:05.569311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:05.569273    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 06:03:05.569589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:05.569563    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 06:03:06.580230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:06.569531    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 06:03:06.580230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:06.569606    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 06:03:06.850280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:06.850147    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:06.879443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:06.879399    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:03:07.572711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:07.572678    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:03:07.574007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:07.573986    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:03:08.772823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:08.772767    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:11.064346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:11.064309    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:12.817198 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:12.817143    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:15.078621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:15.078579    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:16.801479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:16.801441    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:17.172301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:17.172251    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:19.581089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:19.579028    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:20.971815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.971780    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"]
Oct 30 06:03:20.972582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.972563    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c8a35609-ba89-4d0b-9962-d464c2229e3b" containerName="payload-processing"
Oct 30 06:03:20.972582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.972581    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a35609-ba89-4d0b-9962-d464c2229e3b" containerName="payload-processing"
Oct 30 06:03:20.972731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.972591    5250 cpu_manager.go:414] "containerMap: removing container" podUID="2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" containerName="maas-api"
Oct 30 06:03:20.972731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.972596    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" containerName="maas-api"
Oct 30 06:03:20.972845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.972738    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a35609-ba89-4d0b-9962-d464c2229e3b" containerName="payload-processing"
Oct 30 06:03:20.972845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.972746    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a869f1e-d0f0-469f-9f6e-6dd21c35d2d5" containerName="maas-api"
Oct 30 06:03:20.976604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.976579    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:20.992884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.992862    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-1b4b7a\"/\"kube-root-ca.crt\""
Oct 30 06:03:20.996034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:20.996007    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"]
Oct 30 06:03:21.003020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.002981    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1b4b7a\"/\"auth-test-model-1b4b7a-kserve-self-signed-certs\""
Oct 30 06:03:21.003303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.003280    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1b4b7a\"/\"default-dockercfg-wlbgp\""
Oct 30 06:03:21.044776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.044744    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:21.072015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.071973    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24adee0-b7bb-47bc-b838-f7af41449825-tls-certs\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.072258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.072050    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-tmp-dir\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.072258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.072241    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-home\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.072489 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.072471    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-model-cache\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.072582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.072511    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v22f\" (UniqueName: \"kubernetes.io/projected/d24adee0-b7bb-47bc-b838-f7af41449825-kube-api-access-8v22f\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.072685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.072669    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-dshm\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.093045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.093010    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-1b4b7a\"/\"openshift-service-ca.crt\""
Oct 30 06:03:21.176372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.176326    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-dshm\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.176576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.176488    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24adee0-b7bb-47bc-b838-f7af41449825-tls-certs\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.176642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.176577    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-tmp-dir\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.176700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.176681    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-home\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.176852 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.176834    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-model-cache\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.176913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.176883    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v22f\" (UniqueName: \"kubernetes.io/projected/d24adee0-b7bb-47bc-b838-f7af41449825-kube-api-access-8v22f\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.178252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.178212    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-tmp-dir\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.179080 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.179050    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-home\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.179523 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.179385    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-model-cache\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.181861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.181823    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24adee0-b7bb-47bc-b838-f7af41449825-tls-certs\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.193815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.193785    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-dshm\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.240752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.240655    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v22f\" (UniqueName: \"kubernetes.io/projected/d24adee0-b7bb-47bc-b838-f7af41449825-kube-api-access-8v22f\") pod \"auth-test-model-1b4b7a-kserve-9f8697d49-54rth\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.313936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.313901    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:21.748555 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:21.748478    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"]
Oct 30 06:03:21.752123 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:03:21.752097    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd24adee0_b7bb_47bc_b838_f7af41449825.slice/crio-1e7730fc89a632d57be185bf3367e873ed8984953275fad941ae02fc4a1031d6 WatchSource:0}: Error finding container 1e7730fc89a632d57be185bf3367e873ed8984953275fad941ae02fc4a1031d6: Status 404 returned error can't find the container with id 1e7730fc89a632d57be185bf3367e873ed8984953275fad941ae02fc4a1031d6
Oct 30 06:03:22.090384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:22.090257    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:22.448406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:22.448361    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth" event={"ID":"d24adee0-b7bb-47bc-b838-f7af41449825","Type":"ContainerStarted","Data":"a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8"}
Oct 30 06:03:22.448406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:22.448411    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth" event={"ID":"d24adee0-b7bb-47bc-b838-f7af41449825","Type":"ContainerStarted","Data":"1e7730fc89a632d57be185bf3367e873ed8984953275fad941ae02fc4a1031d6"}
Oct 30 06:03:22.490785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:22.490725    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth" podStartSLOduration=2.490710722 podStartE2EDuration="2.490710722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:22.48965595 +0000 UTC m=+1917.320693184" watchObservedRunningTime="2026-10-30 06:03:22.490710722 +0000 UTC m=+1917.321748130"
Oct 30 06:03:23.331108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:23.331078    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:23.693344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:23.693301    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:23.852303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:23.852247    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:24.382277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:24.382232    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:25.956213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:25.953644    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:28.116437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:28.116337    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:30.176920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:30.176873    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:31.330824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:31.330790    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:31.343609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:31.343585    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:31.533493 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:31.533464    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:03:32.140349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:32.140311    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:32.376891 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:03:32.376857    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:03:32.382145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:32.382116    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:32.566338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:32.566248    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:33.305091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:33.305050    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:34.625173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:34.625135    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:34.849114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:34.849053    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:37.126039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:37.126006    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:40.078086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:40.078051    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:40.494900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:40.494864    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:41.785655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:41.785616    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:41.965929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:41.965896    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:42.247203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:42.245747    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:42.604205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:42.604082    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:43.050825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:43.050789    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:43.593277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:43.593170    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:44.313713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:44.313679    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:45.029737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:45.029695    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:47.228427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:47.228384    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:48.258039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:48.258005    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:48.504108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:48.504080    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:49.805655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:49.805621    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:49.932821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:49.932786    5250 generic.go:356] "Generic (PLEG): container finished" podID="b783f198-7706-41b2-8cec-b91b27576482" containerID="d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34" exitCode=0
Oct 30 06:03:49.933009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:49.932819    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" event={"ID":"b783f198-7706-41b2-8cec-b91b27576482","Type":"ContainerDied","Data":"d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34"}
Oct 30 06:03:49.933392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:49.933370    5250 scope.go:122] "RemoveContainer" containerID="d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34"
Oct 30 06:03:49.935774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:49.935745    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:03:49.935862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:49.935818    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:03:49.935905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:49.935859    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:03:50.178770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.178738    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"]
Oct 30 06:03:50.183770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.183746    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:50.213585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.213519    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"]
Oct 30 06:03:50.268948 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.264839    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-080391\""
Oct 30 06:03:50.271325 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.271288    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fbcd02-5042-4a31-9903-ac370777dec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j\" (UID: \"41fbcd02-5042-4a31-9903-ac370777dec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:50.271520 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.271498    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjpf\" (UniqueName: \"kubernetes.io/projected/41fbcd02-5042-4a31-9903-ac370777dec3-kube-api-access-mhjpf\") pod \"payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j\" (UID: \"41fbcd02-5042-4a31-9903-ac370777dec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:50.285920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.285850    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:50.432256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.381419    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fbcd02-5042-4a31-9903-ac370777dec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j\" (UID: \"41fbcd02-5042-4a31-9903-ac370777dec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:50.432805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.432774    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjpf\" (UniqueName: \"kubernetes.io/projected/41fbcd02-5042-4a31-9903-ac370777dec3-kube-api-access-mhjpf\") pod \"payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j\" (UID: \"41fbcd02-5042-4a31-9903-ac370777dec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:50.496050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.496010    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fbcd02-5042-4a31-9903-ac370777dec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j\" (UID: \"41fbcd02-5042-4a31-9903-ac370777dec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:50.551864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.551830    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjpf\" (UniqueName: \"kubernetes.io/projected/41fbcd02-5042-4a31-9903-ac370777dec3-kube-api-access-mhjpf\") pod \"payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j\" (UID: \"41fbcd02-5042-4a31-9903-ac370777dec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:50.800109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.799962    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:50.945520 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:50.945478    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" event={"ID":"b783f198-7706-41b2-8cec-b91b27576482","Type":"ContainerStarted","Data":"74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02"}
Oct 30 06:03:51.083871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:51.083791    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:03:51.085276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:51.085242    5250 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 06:03:51.085426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:51.085319    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 06:03:51.186053 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:03:51.186014    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fbcd02_5042_4a31_9903_ac370777dec3.slice/crio-dba5ccf62f505daf56df29c6b938cbe78527e4d9ac3efc62a7af1f16a9d69f54 WatchSource:0}: Error finding container dba5ccf62f505daf56df29c6b938cbe78527e4d9ac3efc62a7af1f16a9d69f54: Status 404 returned error can't find the container with id dba5ccf62f505daf56df29c6b938cbe78527e4d9ac3efc62a7af1f16a9d69f54
Oct 30 06:03:51.186402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:51.186377    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"]
Oct 30 06:03:51.987286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:51.984369    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j" event={"ID":"41fbcd02-5042-4a31-9903-ac370777dec3","Type":"ContainerStarted","Data":"fedd44354a9aed0e1780d915b8aef565efaa184a0e2382d21a28a9b1524d55ce"}
Oct 30 06:03:51.987286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:51.984433    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j" event={"ID":"41fbcd02-5042-4a31-9903-ac370777dec3","Type":"ContainerStarted","Data":"dba5ccf62f505daf56df29c6b938cbe78527e4d9ac3efc62a7af1f16a9d69f54"}
Oct 30 06:03:51.987286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:51.984482    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:03:52.081066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:52.078330    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j" podStartSLOduration=2.078311205 podStartE2EDuration="2.078311205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:52.076573959 +0000 UTC m=+1946.907611178" watchObservedRunningTime="2026-10-30 06:03:52.078311205 +0000 UTC m=+1946.909348414"
Oct 30 06:03:52.084557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:52.084523    5250 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 06:03:52.084727 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:52.084597    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 06:03:52.084998 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:52.084975    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:03:52.378288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:52.374963    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:53.084042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:53.084006    5250 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 06:03:53.084508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:53.084059    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 06:03:54.084031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:54.083985    5250 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 06:03:54.084266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:54.084039    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 06:03:54.525471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:54.525421    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:55.084693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:55.084655    5250 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 06:03:55.085117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:55.084716    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 06:03:56.084062 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:56.084021    5250 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 06:03:56.084298 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:56.084093    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 06:03:56.686504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:56.686463    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:03:57.089757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:57.089680    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:03:57.090650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:57.090626    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:03:58.538965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:03:58.538832    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:00.480331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:00.480205    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:02.258528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:02.257937    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:02.376403 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:04:02.376356    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:04:04.001648 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:04.001619    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:04:04.393736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:04.393705    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:04.594151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:04.594116    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:06.529092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:06.528072    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:07.090280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.090240    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"]
Oct 30 06:04:07.099318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.099289    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.117710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.117682    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"]
Oct 30 06:04:07.126990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.126961    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d917a1\"/\"default-dockercfg-r62hs\""
Oct 30 06:04:07.126990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.126977    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d917a1\"/\"kube-root-ca.crt\""
Oct 30 06:04:07.127277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.127107    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d917a1\"/\"openshift-service-ca.crt\""
Oct 30 06:04:07.187628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.187592    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:07.195906 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.195884    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d917a1\"/\"test-model-d917a1-kserve-self-signed-certs\""
Oct 30 06:04:07.215216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.215151    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-tmp-dir\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.215397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.215220    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-home\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.215397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.215353    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5kb\" (UniqueName: \"kubernetes.io/projected/fabeb868-afa6-49c3-b089-5905b37f48e4-kube-api-access-ts5kb\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.215559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.215540    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-dshm\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.215609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.215568    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabeb868-afa6-49c3-b089-5905b37f48e4-tls-certs\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.215609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.215602    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-model-cache\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.317889 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.317847    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-tmp-dir\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.318094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.317902    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-home\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.318094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.317963    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5kb\" (UniqueName: \"kubernetes.io/projected/fabeb868-afa6-49c3-b089-5905b37f48e4-kube-api-access-ts5kb\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.318234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.318144    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-dshm\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.318234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.318172    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabeb868-afa6-49c3-b089-5905b37f48e4-tls-certs\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.318361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.318259    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-model-cache\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.318460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.318425    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-tmp-dir\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.318525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.318461    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-home\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.318744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.318716    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-model-cache\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.321150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.321120    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-dshm\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.321421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.321399    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabeb868-afa6-49c3-b089-5905b37f48e4-tls-certs\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.351694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.351584    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5kb\" (UniqueName: \"kubernetes.io/projected/fabeb868-afa6-49c3-b089-5905b37f48e4-kube-api-access-ts5kb\") pod \"test-model-d917a1-kserve-5d9897965-dsdzv\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") " pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.427643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.427604    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:07.856707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:07.856672    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"]
Oct 30 06:04:07.861104 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:04:07.861068    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfabeb868_afa6_49c3_b089_5905b37f48e4.slice/crio-a0f0c0b677b7fe880e5cef7fff1abdc518c8b63f1f37c9992b186bceae21c4df WatchSource:0}: Error finding container a0f0c0b677b7fe880e5cef7fff1abdc518c8b63f1f37c9992b186bceae21c4df: Status 404 returned error can't find the container with id a0f0c0b677b7fe880e5cef7fff1abdc518c8b63f1f37c9992b186bceae21c4df
Oct 30 06:04:08.138422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.138383    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv" event={"ID":"fabeb868-afa6-49c3-b089-5905b37f48e4","Type":"ContainerStarted","Data":"6d574e54cd53149b70519e8b35c050742004d5c4079a270533bbd443407ee5c1"}
Oct 30 06:04:08.138602 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.138436    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv" event={"ID":"fabeb868-afa6-49c3-b089-5905b37f48e4","Type":"ContainerStarted","Data":"a0f0c0b677b7fe880e5cef7fff1abdc518c8b63f1f37c9992b186bceae21c4df"}
Oct 30 06:04:08.196678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.196625    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv" podStartSLOduration=1.196611344 podStartE2EDuration="1.196611344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:08.192459781 +0000 UTC m=+1963.023496998" watchObservedRunningTime="2026-10-30 06:04:08.196611344 +0000 UTC m=+1963.027648560"
Oct 30 06:04:08.437299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.435307    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:08.649760 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.649721    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:08.869357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.869326    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"]
Oct 30 06:04:08.875000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.874977    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:08.891461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.891424    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1dbaf6\"/\"kube-root-ca.crt\""
Oct 30 06:04:08.900073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.900043    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1dbaf6\"/\"test-model-1dbaf6-kserve-self-signed-certs\""
Oct 30 06:04:08.900073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.900077    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1dbaf6\"/\"default-dockercfg-zd58l\""
Oct 30 06:04:08.900302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.900086    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1dbaf6\"/\"openshift-service-ca.crt\""
Oct 30 06:04:08.908614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.908589    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"]
Oct 30 06:04:08.967936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.967895    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-dshm\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:08.968108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.967965    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66283eba-062d-419c-8f0a-449481d5c02b-tls-certs\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:08.968108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.968087    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-tmp-dir\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:08.968339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.968313    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-model-cache\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:08.968404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.968387    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ksb\" (UniqueName: \"kubernetes.io/projected/66283eba-062d-419c-8f0a-449481d5c02b-kube-api-access-p5ksb\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:08.968459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:08.968444    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-home\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.069888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.069803    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-model-cache\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.069888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.069861    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ksb\" (UniqueName: \"kubernetes.io/projected/66283eba-062d-419c-8f0a-449481d5c02b-kube-api-access-p5ksb\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.070138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.069901    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-home\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.070138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.070015    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-dshm\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.070138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.070082    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66283eba-062d-419c-8f0a-449481d5c02b-tls-certs\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.070379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.070146    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-tmp-dir\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.070379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.070331    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-model-cache\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.070447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.070421    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-home\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.070554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.070533    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-tmp-dir\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.072705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.072681    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-dshm\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.073124 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.073101    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66283eba-062d-419c-8f0a-449481d5c02b-tls-certs\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.080855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.080828    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:09.106137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.106108    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ksb\" (UniqueName: \"kubernetes.io/projected/66283eba-062d-419c-8f0a-449481d5c02b-kube-api-access-p5ksb\") pod \"test-model-1dbaf6-kserve-65fcb77d9f-n4ckr\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") " pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.203202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.197491    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:09.656758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:09.656734    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"]
Oct 30 06:04:09.658250 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:04:09.658209    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66283eba_062d_419c_8f0a_449481d5c02b.slice/crio-e57d2e688a83bd78c5bdc634d0bcf98ffe9b3ccc1364bb5b356f32d64755fe86 WatchSource:0}: Error finding container e57d2e688a83bd78c5bdc634d0bcf98ffe9b3ccc1364bb5b356f32d64755fe86: Status 404 returned error can't find the container with id e57d2e688a83bd78c5bdc634d0bcf98ffe9b3ccc1364bb5b356f32d64755fe86
Oct 30 06:04:10.158925 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:10.158889    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr" event={"ID":"66283eba-062d-419c-8f0a-449481d5c02b","Type":"ContainerStarted","Data":"07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34"}
Oct 30 06:04:10.159346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:10.158935    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr" event={"ID":"66283eba-062d-419c-8f0a-449481d5c02b","Type":"ContainerStarted","Data":"e57d2e688a83bd78c5bdc634d0bcf98ffe9b3ccc1364bb5b356f32d64755fe86"}
Oct 30 06:04:10.198538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:10.198510    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:10.221105 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:10.221057    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr" podStartSLOduration=2.221044326 podStartE2EDuration="2.221044326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:10.216969657 +0000 UTC m=+1965.048006877" watchObservedRunningTime="2026-10-30 06:04:10.221044326 +0000 UTC m=+1965.052081564"
Oct 30 06:04:10.879800 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:10.879763    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:13.042607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:13.042556    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:15.040866 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:15.040640    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:15.850095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:15.850062    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:16.040570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:16.040541    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:17.010421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:17.010382    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:17.427819 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:17.427792    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:17.441130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:17.441106    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:18.272921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:18.272891    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:04:19.002770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.002731    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:19.198229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.198187    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:19.210774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.210744    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:19.232252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.232218    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"]
Oct 30 06:04:19.236362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.236340    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.249282 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.249262    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-38cfad\"/\"default-dockercfg-nw6wm\""
Oct 30 06:04:19.250550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.250521    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-38cfad\"/\"kube-root-ca.crt\""
Oct 30 06:04:19.251231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.251210    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-38cfad\"/\"test-model-38cfad-kserve-self-signed-certs\""
Oct 30 06:04:19.252725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.252708    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-38cfad\"/\"openshift-service-ca.crt\""
Oct 30 06:04:19.295237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.295130    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"]
Oct 30 06:04:19.298797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.298772    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:04:19.358616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.358563    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-model-cache\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.358795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.358723    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-dshm\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.359004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.358978    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4pw\" (UniqueName: \"kubernetes.io/projected/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-kube-api-access-gq4pw\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.359083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.359031    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tls-certs\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.359377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.359356    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-home\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.359547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.359412    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tmp-dir\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.361898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.361065    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:19.460912 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.460872    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tmp-dir\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.461106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.460954    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-model-cache\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.461106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.461020    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-dshm\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.461283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.461241    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4pw\" (UniqueName: \"kubernetes.io/projected/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-kube-api-access-gq4pw\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.461353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.461280    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tls-certs\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.461493 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.461469    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tmp-dir\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.461573 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.461557    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-home\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.461974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.461951    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-home\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.462112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.462095    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-model-cache\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.464135 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.464106    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-dshm\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.464802 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.464774    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tls-certs\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.486938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.486910    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4pw\" (UniqueName: \"kubernetes.io/projected/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-kube-api-access-gq4pw\") pod \"test-model-38cfad-kserve-68d7685f65-46k7f\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") " pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:19.554089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:19.554003    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:20.025391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:20.025366    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"]
Oct 30 06:04:20.026812 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:04:20.026783    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcec2480_ff5a_4368_ac2f_aeaf7f1c7b0e.slice/crio-136e4a3742e4eaa66fc3aab861798fa55d97d3a3295ad1bfce074d45aaa7319c WatchSource:0}: Error finding container 136e4a3742e4eaa66fc3aab861798fa55d97d3a3295ad1bfce074d45aaa7319c: Status 404 returned error can't find the container with id 136e4a3742e4eaa66fc3aab861798fa55d97d3a3295ad1bfce074d45aaa7319c
Oct 30 06:04:20.295446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:20.295360    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f" event={"ID":"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e","Type":"ContainerStarted","Data":"9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03"}
Oct 30 06:04:20.295446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:20.295412    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f" event={"ID":"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e","Type":"ContainerStarted","Data":"136e4a3742e4eaa66fc3aab861798fa55d97d3a3295ad1bfce074d45aaa7319c"}
Oct 30 06:04:20.335979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:20.335931    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f" podStartSLOduration=1.335915931 podStartE2EDuration="1.335915931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:20.333204881 +0000 UTC m=+1975.164242089" watchObservedRunningTime="2026-10-30 06:04:20.335915931 +0000 UTC m=+1975.166953147"
Oct 30 06:04:20.555586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:20.555489    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:20.951433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:20.946704    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:21.397060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:21.396980    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:23.165203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:23.162737    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:25.005119 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:25.005083    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:26.920936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:26.920895    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:28.991844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:28.991804    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:29.554712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:29.554680    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:29.567831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:29.567806    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:30.104561 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:30.104526    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:30.429225 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:30.425513    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:04:30.937284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:30.937244    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:32.384672 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:04:32.384626    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:04:32.825979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:32.825874    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:34.644694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:34.644662    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:34.886712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:34.886674    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:35.208837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:35.208802    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:36.302265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:36.300912    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:36.618545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:36.618447    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:37.448197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.448147    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"]
Oct 30 06:04:37.453426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.453407    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.475886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.475857    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"]
Oct 30 06:04:37.477962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.477941    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:04:37.478093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.478007    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:04:37.513415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.513374    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.513701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.513672    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkjk\" (UniqueName: \"kubernetes.io/projected/3f8e890c-5572-4462-90a0-06733f199fbb-kube-api-access-pmkjk\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.513825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.513803    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.615666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.615630    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkjk\" (UniqueName: \"kubernetes.io/projected/3f8e890c-5572-4462-90a0-06733f199fbb-kube-api-access-pmkjk\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.615867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.615705    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.615867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.615851    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.616552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.616524    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.618294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.618275    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.637584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.637562    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkjk\" (UniqueName: \"kubernetes.io/projected/3f8e890c-5572-4462-90a0-06733f199fbb-kube-api-access-pmkjk\") pod \"maas-api-revoke-sub-e77c3e5b-cht6f\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:37.768574 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:37.768495    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:38.086208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:38.086147    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"]
Oct 30 06:04:38.087254 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:04:38.087229    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8e890c_5572_4462_90a0_06733f199fbb.slice/crio-65acb75c62af350b4d8904428778ff22086a1cda2b0f36e17104d06690d1754d WatchSource:0}: Error finding container 65acb75c62af350b4d8904428778ff22086a1cda2b0f36e17104d06690d1754d: Status 404 returned error can't find the container with id 65acb75c62af350b4d8904428778ff22086a1cda2b0f36e17104d06690d1754d
Oct 30 06:04:38.462022 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:38.461986    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f" event={"ID":"3f8e890c-5572-4462-90a0-06733f199fbb","Type":"ContainerStarted","Data":"65acb75c62af350b4d8904428778ff22086a1cda2b0f36e17104d06690d1754d"}
Oct 30 06:04:38.588156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:38.588114    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:39.471528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:39.471477    5250 generic.go:356] "Generic (PLEG): container finished" podID="3f8e890c-5572-4462-90a0-06733f199fbb" containerID="438741761fc3d6ddda37c87cb75703b5a689ec0bd40d517114327f671b0de418" exitCode=0
Oct 30 06:04:39.471926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:39.471553    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f" event={"ID":"3f8e890c-5572-4462-90a0-06733f199fbb","Type":"ContainerDied","Data":"438741761fc3d6ddda37c87cb75703b5a689ec0bd40d517114327f671b0de418"}
Oct 30 06:04:40.586408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.586368    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:40.697521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.697491    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:40.735939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.735825    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:40.768324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.768280    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8e890c-5572-4462-90a0-06733f199fbb-kube-api-access-pmkjk\" (UniqueName: \"kubernetes.io/projected/3f8e890c-5572-4462-90a0-06733f199fbb-kube-api-access-pmkjk\") pod \"3f8e890c-5572-4462-90a0-06733f199fbb\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") "
Oct 30 06:04:40.768530 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.768479    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-xks-ca\") pod \"3f8e890c-5572-4462-90a0-06733f199fbb\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") "
Oct 30 06:04:40.768610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.768561    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-openshift-ca\") pod \"3f8e890c-5572-4462-90a0-06733f199fbb\" (UID: \"3f8e890c-5572-4462-90a0-06733f199fbb\") "
Oct 30 06:04:40.770737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.770705    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-openshift-ca" pod "3f8e890c-5572-4462-90a0-06733f199fbb" (UID: "3f8e890c-5572-4462-90a0-06733f199fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:04:40.772390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.772359    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8e890c-5572-4462-90a0-06733f199fbb-kube-api-access-pmkjk" pod "3f8e890c-5572-4462-90a0-06733f199fbb" (UID: "3f8e890c-5572-4462-90a0-06733f199fbb"). InnerVolumeSpecName "kube-api-access-pmkjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:04:40.772842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.772799    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-xks-ca" pod "3f8e890c-5572-4462-90a0-06733f199fbb" (UID: "3f8e890c-5572-4462-90a0-06733f199fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:04:40.872195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.872147    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkjk\" (UniqueName: \"kubernetes.io/projected/3f8e890c-5572-4462-90a0-06733f199fbb-kube-api-access-pmkjk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:04:40.872195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.872197    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:04:40.872439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:40.872208    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e890c-5572-4462-90a0-06733f199fbb-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:04:41.072450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:41.072350    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:41.496458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:41.496428    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"
Oct 30 06:04:41.496666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:41.496460    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f" event={"ID":"3f8e890c-5572-4462-90a0-06733f199fbb","Type":"ContainerDied","Data":"65acb75c62af350b4d8904428778ff22086a1cda2b0f36e17104d06690d1754d"}
Oct 30 06:04:41.496666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:41.496522    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65acb75c62af350b4d8904428778ff22086a1cda2b0f36e17104d06690d1754d"
Oct 30 06:04:42.669454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:42.669383    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:44.732952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:44.732903    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:46.599753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:46.599714    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:48.607027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:48.606993    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:50.353140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:50.353068    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:52.367279 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:52.367241    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:54.236908 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:54.236862    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:56.211301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:56.211267    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:04:58.115964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:04:58.115900    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:00.043991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:00.043954    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:01.169066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:01.169030    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"]
Oct 30 06:05:01.169505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:01.169278    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy" containerID="cri-o://3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b" gracePeriod=30
Oct 30 06:05:02.393655 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:05:02.393611    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:05:03.178090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:03.178053    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:05.144332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.144297    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"]
Oct 30 06:05:05.273901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.273840    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:05.434027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.433926    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"]
Oct 30 06:05:05.434931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.434891    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy" containerID="cri-o://795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986" gracePeriod=30
Oct 30 06:05:05.522981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.522952    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"]
Oct 30 06:05:05.523230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.523154    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" containerID="cri-o://e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915" gracePeriod=30
Oct 30 06:05:05.884674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.884642    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"]
Oct 30 06:05:05.885525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.885506    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3f8e890c-5572-4462-90a0-06733f199fbb" containerName="revoke-keys"
Oct 30 06:05:05.885525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.885525    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8e890c-5572-4462-90a0-06733f199fbb" containerName="revoke-keys"
Oct 30 06:05:05.885689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.885679    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8e890c-5572-4462-90a0-06733f199fbb" containerName="revoke-keys"
Oct 30 06:05:05.891670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.891650    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:05.930421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.911711    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"]
Oct 30 06:05:05.930421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.922404    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:05:05.930421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.922577    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:05:05.976875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.976838    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr4kl\" (UniqueName: \"kubernetes.io/projected/442c0652-4a4e-4e28-9422-490463c5eed4-kube-api-access-zr4kl\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:05.977087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.977041    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:05.977087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.977080    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:05.980254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.980202    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"]
Oct 30 06:05:05.980527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:05.980493    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy" containerID="cri-o://baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a" gracePeriod=30
Oct 30 06:05:06.078945 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.078914    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr4kl\" (UniqueName: \"kubernetes.io/projected/442c0652-4a4e-4e28-9422-490463c5eed4-kube-api-access-zr4kl\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:06.079145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.079047    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:06.079145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.079078    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:06.079717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.079691    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:06.081854 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.081832    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:06.109997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.109965    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr4kl\" (UniqueName: \"kubernetes.io/projected/442c0652-4a4e-4e28-9422-490463c5eed4-kube-api-access-zr4kl\") pod \"maas-api-revoke-sub-fbf6f60e-c45cp\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:06.241526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.241487    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:06.498348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.498127    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:06.623221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.623156    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:05:06.646130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.646103    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"]
Oct 30 06:05:06.647545 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:05:06.647518    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442c0652_4a4e_4e28_9422_490463c5eed4.slice/crio-3cae1df7c023a46779b5c10896ec3d15545e11aa12a2233a1dacc17f22a1d408 WatchSource:0}: Error finding container 3cae1df7c023a46779b5c10896ec3d15545e11aa12a2233a1dacc17f22a1d408: Status 404 returned error can't find the container with id 3cae1df7c023a46779b5c10896ec3d15545e11aa12a2233a1dacc17f22a1d408
Oct 30 06:05:06.714099 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714065    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-token\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-token\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.714363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714144    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-envoy\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.714363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714285    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872e7a34-3fe4-4ae5-b29b-26848745a69c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872e7a34-3fe4-4ae5-b29b-26848745a69c-istiod-ca-cert\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.714363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714349    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-certs\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.714555 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714481    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-socket\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.714623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714517    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-kube-api-access-ccdmm\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-kube-api-access-ccdmm\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.714725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714704    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-credential-socket\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.714826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714799    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-podinfo\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.714826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714816    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-certs" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:06.714955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.714861    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-data\") pod \"872e7a34-3fe4-4ae5-b29b-26848745a69c\" (UID: \"872e7a34-3fe4-4ae5-b29b-26848745a69c\") "
Oct 30 06:05:06.715634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.715451    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872e7a34-3fe4-4ae5-b29b-26848745a69c-istiod-ca-cert" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:05:06.715634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.715450    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-credential-socket" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:06.715817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.715648    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-data" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:06.715817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.715710    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.715817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.715730    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872e7a34-3fe4-4ae5-b29b-26848745a69c-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.715817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.715744    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.721051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.720780    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-token" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:06.721051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.720847    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-envoy" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:06.721051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.720865    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-socket" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:06.721051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.720909    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-podinfo" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:05:06.722165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.722136    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-kube-api-access-ccdmm" pod "872e7a34-3fe4-4ae5-b29b-26848745a69c" (UID: "872e7a34-3fe4-4ae5-b29b-26848745a69c"). InnerVolumeSpecName "kube-api-access-ccdmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:06.817853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.817760    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.817853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.817805    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdmm\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-kube-api-access-ccdmm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.817853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.817822    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.817853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.817837    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.817853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.817850    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.818138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.817862    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872e7a34-3fe4-4ae5-b29b-26848745a69c-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:06.841508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.841469    5250 generic.go:356] "Generic (PLEG): container finished" podID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerID="3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b" exitCode=0
Oct 30 06:05:06.841686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.841518    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"
Oct 30 06:05:06.841686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.841559    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" event={"ID":"872e7a34-3fe4-4ae5-b29b-26848745a69c","Type":"ContainerDied","Data":"3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b"}
Oct 30 06:05:06.841686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.841621    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z" event={"ID":"872e7a34-3fe4-4ae5-b29b-26848745a69c","Type":"ContainerDied","Data":"ba8ae0b94c1835c5cac5d02d55c7f1f24992c1ac544069e17eb56875d4dd2bac"}
Oct 30 06:05:06.841686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.841652    5250 scope.go:122] "RemoveContainer" containerID="3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b"
Oct 30 06:05:06.843191 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.843150    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp" event={"ID":"442c0652-4a4e-4e28-9422-490463c5eed4","Type":"ContainerStarted","Data":"3cae1df7c023a46779b5c10896ec3d15545e11aa12a2233a1dacc17f22a1d408"}
Oct 30 06:05:06.858642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.858613    5250 scope.go:122] "RemoveContainer" containerID="3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b"
Oct 30 06:05:06.858949 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:05:06.858927    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b\": container with ID starting with 3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b not found: ID does not exist" containerID="3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b"
Oct 30 06:05:06.859049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.858955    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b"} err="failed to get container status \"3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b\": rpc error: code = NotFound desc = could not find container \"3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b\": container with ID starting with 3ac65043650e8b3851d7fc696ecf63ff09ac34d0ca4bbe20f54b7c5bb78a503b not found: ID does not exist"
Oct 30 06:05:06.911438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.911404    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"]
Oct 30 06:05:06.926213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:06.926167    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-cgf2z"]
Oct 30 06:05:07.169255 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:07.169157    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"]
Oct 30 06:05:07.169489 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:07.169449    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy" containerID="cri-o://5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447" gracePeriod=30
Oct 30 06:05:07.297541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:07.297498    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" path="/var/lib/kubelet/pods/872e7a34-3fe4-4ae5-b29b-26848745a69c/volumes"
Oct 30 06:05:07.855744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:07.855707    5250 generic.go:356] "Generic (PLEG): container finished" podID="442c0652-4a4e-4e28-9422-490463c5eed4" containerID="b57a5f4b4af6f12885b10f2dda1ad21237021eb8f844c74f2bc8a08f6eb5e21f" exitCode=0
Oct 30 06:05:07.855904 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:07.855792    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp" event={"ID":"442c0652-4a4e-4e28-9422-490463c5eed4","Type":"ContainerDied","Data":"b57a5f4b4af6f12885b10f2dda1ad21237021eb8f844c74f2bc8a08f6eb5e21f"}
Oct 30 06:05:08.378021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:08.377878    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:09.042039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.042010    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:09.158655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.158623    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-xks-ca\") pod \"442c0652-4a4e-4e28-9422-490463c5eed4\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") "
Oct 30 06:05:09.158862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.158722    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442c0652-4a4e-4e28-9422-490463c5eed4-kube-api-access-zr4kl\" (UniqueName: \"kubernetes.io/projected/442c0652-4a4e-4e28-9422-490463c5eed4-kube-api-access-zr4kl\") pod \"442c0652-4a4e-4e28-9422-490463c5eed4\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") "
Oct 30 06:05:09.159027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.159008    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-openshift-ca\") pod \"442c0652-4a4e-4e28-9422-490463c5eed4\" (UID: \"442c0652-4a4e-4e28-9422-490463c5eed4\") "
Oct 30 06:05:09.159448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.159308    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-openshift-ca" pod "442c0652-4a4e-4e28-9422-490463c5eed4" (UID: "442c0652-4a4e-4e28-9422-490463c5eed4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:05:09.159783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.159762    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:09.161049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.161026    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-xks-ca" pod "442c0652-4a4e-4e28-9422-490463c5eed4" (UID: "442c0652-4a4e-4e28-9422-490463c5eed4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:05:09.161200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.161148    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442c0652-4a4e-4e28-9422-490463c5eed4-kube-api-access-zr4kl" pod "442c0652-4a4e-4e28-9422-490463c5eed4" (UID: "442c0652-4a4e-4e28-9422-490463c5eed4"). InnerVolumeSpecName "kube-api-access-zr4kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:09.261711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.261679    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442c0652-4a4e-4e28-9422-490463c5eed4-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:09.261711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.261707    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr4kl\" (UniqueName: \"kubernetes.io/projected/442c0652-4a4e-4e28-9422-490463c5eed4-kube-api-access-zr4kl\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:09.874541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.874512    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"
Oct 30 06:05:09.874930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.874552    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp" event={"ID":"442c0652-4a4e-4e28-9422-490463c5eed4","Type":"ContainerDied","Data":"3cae1df7c023a46779b5c10896ec3d15545e11aa12a2233a1dacc17f22a1d408"}
Oct 30 06:05:09.874930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:09.874605    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cae1df7c023a46779b5c10896ec3d15545e11aa12a2233a1dacc17f22a1d408"
Oct 30 06:05:10.302716 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.302678    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:10.331978 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.331923    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:05:10.332160 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.331997    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:05:10.753778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.753746    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 06:05:10.831212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.827766    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 06:05:10.889990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.889953    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-istio-token\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-istio-token\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.890066    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3548b95a-7ae2-45f9-896a-5b9499810632-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b95a-7ae2-45f9-896a-5b9499810632-istio-podinfo\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.890118    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-data\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.890218    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3548b95a-7ae2-45f9-896a-5b9499810632-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b95a-7ae2-45f9-896a-5b9499810632-istiod-ca-cert\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.890277    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-kube-api-access-956vz\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-kube-api-access-956vz\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.890441    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-credential-socket\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.890617    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-socket\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.890665    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-envoy\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.890727    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-certs\") pod \"3548b95a-7ae2-45f9-896a-5b9499810632\" (UID: \"3548b95a-7ae2-45f9-896a-5b9499810632\") "
Oct 30 06:05:10.891609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.891392    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3548b95a-7ae2-45f9-896a-5b9499810632-istiod-ca-cert" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:05:10.891670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.891631    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-certs" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.892263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.891766    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b95a-7ae2-45f9-896a-5b9499810632-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.892263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.891837    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.892263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.892229    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-data" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.892645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.892462    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-socket" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.892645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.892517    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-credential-socket" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.894932 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.894905    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3548b95a-7ae2-45f9-896a-5b9499810632-istio-podinfo" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:05:10.895076 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.895059    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-envoy" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.895474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.895445    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-istio-token" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:10.895575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.895477    5250 generic.go:356] "Generic (PLEG): container finished" podID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerID="e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915" exitCode=0
Oct 30 06:05:10.895575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.895509    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" event={"ID":"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b","Type":"ContainerDied","Data":"e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915"}
Oct 30 06:05:10.895575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.895534    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"
Oct 30 06:05:10.895728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.895614    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l" event={"ID":"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b","Type":"ContainerDied","Data":"e79f78737f26ebe6d049d93fb0a2be01f4e61bf70dd8134b64dc72c2a3d89752"}
Oct 30 06:05:10.895728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.895646    5250 scope.go:122] "RemoveContainer" containerID="e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915"
Oct 30 06:05:10.895804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.895754    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-kube-api-access-956vz" pod "3548b95a-7ae2-45f9-896a-5b9499810632" (UID: "3548b95a-7ae2-45f9-896a-5b9499810632"). InnerVolumeSpecName "kube-api-access-956vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:10.898296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.898271    5250 generic.go:356] "Generic (PLEG): container finished" podID="3548b95a-7ae2-45f9-896a-5b9499810632" containerID="795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986" exitCode=0
Oct 30 06:05:10.898400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.898341    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"
Oct 30 06:05:10.898465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.898341    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" event={"ID":"3548b95a-7ae2-45f9-896a-5b9499810632","Type":"ContainerDied","Data":"795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986"}
Oct 30 06:05:10.898514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.898471    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn" event={"ID":"3548b95a-7ae2-45f9-896a-5b9499810632","Type":"ContainerDied","Data":"6dc8cc72fd9829f8a10cdd317ae0c46de5be0d0ad0cb6a7af1c2d4dd6e7a6712"}
Oct 30 06:05:10.912481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.912463    5250 scope.go:122] "RemoveContainer" containerID="e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915"
Oct 30 06:05:10.912843 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:05:10.912820    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915\": container with ID starting with e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915 not found: ID does not exist" containerID="e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915"
Oct 30 06:05:10.912963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.912850    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915"} err="failed to get container status \"e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915\": rpc error: code = NotFound desc = could not find container \"e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915\": container with ID starting with e9e9d2a67a4b7b862236b143693ad781623ef59a03d9dd1fe589c42fc6e66915 not found: ID does not exist"
Oct 30 06:05:10.912963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.912869    5250 scope.go:122] "RemoveContainer" containerID="795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986"
Oct 30 06:05:10.931426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.931408    5250 scope.go:122] "RemoveContainer" containerID="795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986"
Oct 30 06:05:10.931909 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:05:10.931883    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986\": container with ID starting with 795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986 not found: ID does not exist" containerID="795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986"
Oct 30 06:05:10.931999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.931920    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986"} err="failed to get container status \"795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986\": rpc error: code = NotFound desc = could not find container \"795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986\": container with ID starting with 795ee5a8bc0e34c780c7f0e019718507e7db90de42e3fc13c9e511b03c9da986 not found: ID does not exist"
Oct 30 06:05:10.961458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.961434    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"]
Oct 30 06:05:10.968273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.968250    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-bmfdn"]
Oct 30 06:05:10.992977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.992951    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-certs\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.993087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.992997    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-data\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.993087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993070    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-envoy\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.993204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993106    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istiod-ca-cert\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.993204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993141    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-kube-api-access-xkl7z\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-kube-api-access-xkl7z\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.993335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993284    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-certs" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.993335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993311    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-token\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-token\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.993443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993372    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-podinfo\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.993443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993414    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-socket\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.993443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993438    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-credential-socket\") pod \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\" (UID: \"0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b\") "
Oct 30 06:05:10.994117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.993940    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-socket" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.994211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994134    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-credential-socket" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994446    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994470    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b95a-7ae2-45f9-896a-5b9499810632-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994483    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994496    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-956vz\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-kube-api-access-956vz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994509    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994521    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994533    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994543    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994555    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b95a-7ae2-45f9-896a-5b9499810632-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.994606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994567    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b95a-7ae2-45f9-896a-5b9499810632-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:10.995310 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994773    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istiod-ca-cert" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:05:10.995310 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.994810    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-data" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.995931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.995905    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-token" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:10.996017 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.995943    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-kube-api-access-xkl7z" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "kube-api-access-xkl7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:10.996062 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.996033    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-envoy" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:10.996731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:10.996714    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-podinfo" pod "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" (UID: "0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:05:11.096424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.096353    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.096424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.096382    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.096424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.096391    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.096424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.096399    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.096424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.096406    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkl7z\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-kube-api-access-xkl7z\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.096424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.096415    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.252467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.252315    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"]
Oct 30 06:05:11.256850 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.256818    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-sr56l"]
Oct 30 06:05:11.288738 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.288711    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 06:05:11.292078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.292053    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" path="/var/lib/kubelet/pods/0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b/volumes"
Oct 30 06:05:11.292600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.292584    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" path="/var/lib/kubelet/pods/3548b95a-7ae2-45f9-896a-5b9499810632/volumes"
Oct 30 06:05:11.401006 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.400980    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-kube-api-access-87b4d\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-kube-api-access-87b4d\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401021    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istiod-ca-cert\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401047    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-token\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-token\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401098    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-data\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401151    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-socket\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401203    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-certs\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401342    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-envoy\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401450    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-podinfo\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401476    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-credential-socket\") pod \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\" (UID: \"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214\") "
Oct 30 06:05:11.401495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401474    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-data" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:11.401707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401492    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istiod-ca-cert" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:05:11.401762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401719    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-certs" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:11.401762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.401732    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-socket" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:11.403006 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.402971    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.403762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.403661    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.403762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.403687    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.403762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.403701    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.404016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.403803    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-credential-socket" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:11.404253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.404231    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-kube-api-access-87b4d" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "kube-api-access-87b4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:11.404339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.404268    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-podinfo" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:05:11.404402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.404387    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-token" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:11.404577 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.404561    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-envoy" pod "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" (UID: "c4ae5ad1-1ee9-4a65-a10f-72004c9ea214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:11.505341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.505313    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.505341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.505339    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.505558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.505349    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87b4d\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-kube-api-access-87b4d\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.505558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.505358    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.505558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.505368    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:11.911031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.910992    5250 generic.go:356] "Generic (PLEG): container finished" podID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerID="baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a" exitCode=0
Oct 30 06:05:11.911468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.911044    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"
Oct 30 06:05:11.911468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.911057    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" event={"ID":"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214","Type":"ContainerDied","Data":"baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a"}
Oct 30 06:05:11.911468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.911098    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g" event={"ID":"c4ae5ad1-1ee9-4a65-a10f-72004c9ea214","Type":"ContainerDied","Data":"a3c2d35cdefeb71a4fa6a1cbf2ac8b6ac838a9869fcd20455a0930aad656c7b7"}
Oct 30 06:05:11.911468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.911121    5250 scope.go:122] "RemoveContainer" containerID="baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a"
Oct 30 06:05:11.923283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.923264    5250 scope.go:122] "RemoveContainer" containerID="baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a"
Oct 30 06:05:11.923568 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:05:11.923545    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a\": container with ID starting with baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a not found: ID does not exist" containerID="baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a"
Oct 30 06:05:11.923654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.923575    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a"} err="failed to get container status \"baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a\": rpc error: code = NotFound desc = could not find container \"baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a\": container with ID starting with baffbcc4024e45431b08abd0fd339164d884ede4f530941bcc0f797e8d477e5a not found: ID does not exist"
Oct 30 06:05:11.965063 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.965035    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"]
Oct 30 06:05:11.973806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:11.973785    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-kks5g"]
Oct 30 06:05:12.330846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.329806    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:12.516359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.516329    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 06:05:12.524940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.524908    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-credential-socket\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.525171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.524953    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istiod-ca-cert\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.525171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525067    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-data\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.525171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525216    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-credential-socket" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:12.525436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525332    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-data" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:12.525575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525552    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-certs" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:12.525611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525257    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-certs\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.525669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525653    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-podinfo\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.525713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525690    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-envoy\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.525793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525776    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-token\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-token\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.525852 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525831    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-socket\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.525910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.525873    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-kube-api-access-j5cc7\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-kube-api-access-j5cc7\") pod \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\" (UID: \"4aa3bcb2-8640-47d5-a423-fda32c2d77d9\") "
Oct 30 06:05:12.526421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.526369    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-socket" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:12.527158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.527134    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.527288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.527162    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.527288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.527220    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.527288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.527233    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.527540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.527516    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istiod-ca-cert" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:05:12.528445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.528420    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-envoy" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:12.528733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.528709    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-token" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:12.528733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.528710    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-podinfo" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:05:12.529392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.529365    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-kube-api-access-j5cc7" pod "4aa3bcb2-8640-47d5-a423-fda32c2d77d9" (UID: "4aa3bcb2-8640-47d5-a423-fda32c2d77d9"). InnerVolumeSpecName "kube-api-access-j5cc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:12.628822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.628775    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.628822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.628822    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.629081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.628835    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.629081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.628847    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.629081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.628861    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5cc7\" (UniqueName: \"kubernetes.io/projected/4aa3bcb2-8640-47d5-a423-fda32c2d77d9-kube-api-access-j5cc7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:12.922088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.922051    5250 generic.go:356] "Generic (PLEG): container finished" podID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerID="5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447" exitCode=0
Oct 30 06:05:12.922544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.922111    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"
Oct 30 06:05:12.922544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.922109    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" event={"ID":"4aa3bcb2-8640-47d5-a423-fda32c2d77d9","Type":"ContainerDied","Data":"5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447"}
Oct 30 06:05:12.922544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.922252    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j" event={"ID":"4aa3bcb2-8640-47d5-a423-fda32c2d77d9","Type":"ContainerDied","Data":"58589c07a38317ec685fece36c70f69d56000774ad375a890a7210ee3defa8a1"}
Oct 30 06:05:12.922544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.922298    5250 scope.go:122] "RemoveContainer" containerID="5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447"
Oct 30 06:05:12.937534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.937513    5250 scope.go:122] "RemoveContainer" containerID="5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447"
Oct 30 06:05:12.937889 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:05:12.937856    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447\": container with ID starting with 5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447 not found: ID does not exist" containerID="5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447"
Oct 30 06:05:12.937940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:12.937905    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447"} err="failed to get container status \"5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447\": rpc error: code = NotFound desc = could not find container \"5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447\": container with ID starting with 5a71ecb93ba78f9d6764cc86ffd92e1185dc333c1e74488767b532e0247b5447 not found: ID does not exist"
Oct 30 06:05:13.007963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:13.007930    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"]
Oct 30 06:05:13.020388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:13.020356    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-pkd9j"]
Oct 30 06:05:13.294786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:13.294702    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" path="/var/lib/kubelet/pods/4aa3bcb2-8640-47d5-a423-fda32c2d77d9/volumes"
Oct 30 06:05:13.295478 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:13.295459    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" path="/var/lib/kubelet/pods/c4ae5ad1-1ee9-4a65-a10f-72004c9ea214/volumes"
Oct 30 06:05:14.494493 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:14.494440    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:16.462335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:16.462299    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:18.585400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:18.585369    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:20.175855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.175815    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl" podUID="0e7991cb-1a59-46de-8fa8-c228c7e4033d" containerName="main" containerID="cri-o://607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91" gracePeriod=45
Oct 30 06:05:20.503437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.503373    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:05:20.557498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.557407    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:20.652232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652198    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-model-cache\") pod \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") "
Oct 30 06:05:20.652456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652304    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-home\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-home\") pod \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") "
Oct 30 06:05:20.652456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652334    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-dshm\") pod \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") "
Oct 30 06:05:20.652456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652362    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tmp-dir\") pod \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") "
Oct 30 06:05:20.652626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652515    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-model-cache" pod "0e7991cb-1a59-46de-8fa8-c228c7e4033d" (UID: "0e7991cb-1a59-46de-8fa8-c228c7e4033d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:20.652626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652581    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tls-certs\") pod \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") "
Oct 30 06:05:20.652626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652616    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7991cb-1a59-46de-8fa8-c228c7e4033d-kube-api-access-mx9rb\" (UniqueName: \"kubernetes.io/projected/0e7991cb-1a59-46de-8fa8-c228c7e4033d-kube-api-access-mx9rb\") pod \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\" (UID: \"0e7991cb-1a59-46de-8fa8-c228c7e4033d\") "
Oct 30 06:05:20.652775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652617    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-home" pod "0e7991cb-1a59-46de-8fa8-c228c7e4033d" (UID: "0e7991cb-1a59-46de-8fa8-c228c7e4033d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:20.652775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.652693    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tmp-dir" pod "0e7991cb-1a59-46de-8fa8-c228c7e4033d" (UID: "0e7991cb-1a59-46de-8fa8-c228c7e4033d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:20.653129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.653110    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:20.653129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.653130    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:20.653526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.653143    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:20.654817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.654787    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-dshm" pod "0e7991cb-1a59-46de-8fa8-c228c7e4033d" (UID: "0e7991cb-1a59-46de-8fa8-c228c7e4033d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:20.654940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.654857    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tls-certs" pod "0e7991cb-1a59-46de-8fa8-c228c7e4033d" (UID: "0e7991cb-1a59-46de-8fa8-c228c7e4033d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:05:20.654940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.654904    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7991cb-1a59-46de-8fa8-c228c7e4033d-kube-api-access-mx9rb" pod "0e7991cb-1a59-46de-8fa8-c228c7e4033d" (UID: "0e7991cb-1a59-46de-8fa8-c228c7e4033d"). InnerVolumeSpecName "kube-api-access-mx9rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:20.754962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.754885    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7991cb-1a59-46de-8fa8-c228c7e4033d-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:20.754962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.754914    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx9rb\" (UniqueName: \"kubernetes.io/projected/0e7991cb-1a59-46de-8fa8-c228c7e4033d-kube-api-access-mx9rb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:20.754962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.754924    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7991cb-1a59-46de-8fa8-c228c7e4033d-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:20.782691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.782658    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"]
Oct 30 06:05:20.782909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.782886    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy" containerID="cri-o://ec0ce2d400890f81b8ca279a12faadc381d8c14825a1c06e48f37061fd074a9a" gracePeriod=30
Oct 30 06:05:20.993543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.993512    5250 generic.go:356] "Generic (PLEG): container finished" podID="0e7991cb-1a59-46de-8fa8-c228c7e4033d" containerID="607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91" exitCode=0
Oct 30 06:05:20.993736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.993559    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"
Oct 30 06:05:20.993736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.993598    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl" event={"ID":"0e7991cb-1a59-46de-8fa8-c228c7e4033d","Type":"ContainerDied","Data":"607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91"}
Oct 30 06:05:20.993736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.993648    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl" event={"ID":"0e7991cb-1a59-46de-8fa8-c228c7e4033d","Type":"ContainerDied","Data":"3ea0c905337e6a517d920a114fa58d01aac206fd11238d5ccbd754bc6e4715ed"}
Oct 30 06:05:20.993736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:20.993671    5250 scope.go:122] "RemoveContainer" containerID="607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91"
Oct 30 06:05:21.008065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.008039    5250 scope.go:122] "RemoveContainer" containerID="607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91"
Oct 30 06:05:21.008492 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:05:21.008462    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91\": container with ID starting with 607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91 not found: ID does not exist" containerID="607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91"
Oct 30 06:05:21.008609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.008504    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91"} err="failed to get container status \"607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91\": rpc error: code = NotFound desc = could not find container \"607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91\": container with ID starting with 607482754a675d8f6e2737fe2ad58ebbaa7eb63d765eb38705f8a2f8d94dca91 not found: ID does not exist"
Oct 30 06:05:21.051750 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.051715    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"]
Oct 30 06:05:21.056133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.056107    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl"]
Oct 30 06:05:21.283315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.283240    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7991cb-1a59-46de-8fa8-c228c7e4033d" path="/var/lib/kubelet/pods/0e7991cb-1a59-46de-8fa8-c228c7e4033d/volumes"
Oct 30 06:05:21.413783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.413744    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0382c2/test-model-0382c2-kserve-56b886478c-lxzrl" podUID="0e7991cb-1a59-46de-8fa8-c228c7e4033d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:05:21.626473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.626376    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"]
Oct 30 06:05:21.628445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628294    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy"
Oct 30 06:05:21.628445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628321    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy"
Oct 30 06:05:21.628445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628349    5250 cpu_manager.go:414] "containerMap: removing container" podUID="442c0652-4a4e-4e28-9422-490463c5eed4" containerName="revoke-keys"
Oct 30 06:05:21.628445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628357    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442c0652-4a4e-4e28-9422-490463c5eed4" containerName="revoke-keys"
Oct 30 06:05:21.628445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628409    5250 cpu_manager.go:414] "containerMap: removing container" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy"
Oct 30 06:05:21.628445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628418    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy"
Oct 30 06:05:21.628445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628448    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy"
Oct 30 06:05:21.628445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628455    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy"
Oct 30 06:05:21.629010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628537    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0e7991cb-1a59-46de-8fa8-c228c7e4033d" containerName="main"
Oct 30 06:05:21.629010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628544    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7991cb-1a59-46de-8fa8-c228c7e4033d" containerName="main"
Oct 30 06:05:21.629010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628568    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy"
Oct 30 06:05:21.629010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628575    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy"
Oct 30 06:05:21.629010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628596    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy"
Oct 30 06:05:21.629010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628602    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy"
Oct 30 06:05:21.629010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628977    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="872e7a34-3fe4-4ae5-b29b-26848745a69c" containerName="istio-proxy"
Oct 30 06:05:21.629010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.628995    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0596f462-3f2b-4f5b-8c22-10ccd7ad5a8b" containerName="istio-proxy"
Oct 30 06:05:21.629409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.629030    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7991cb-1a59-46de-8fa8-c228c7e4033d" containerName="main"
Oct 30 06:05:21.629409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.629041    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa3bcb2-8640-47d5-a423-fda32c2d77d9" containerName="istio-proxy"
Oct 30 06:05:21.629409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.629055    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3548b95a-7ae2-45f9-896a-5b9499810632" containerName="istio-proxy"
Oct 30 06:05:21.629409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.629067    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ae5ad1-1ee9-4a65-a10f-72004c9ea214" containerName="istio-proxy"
Oct 30 06:05:21.629409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.629083    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="442c0652-4a4e-4e28-9422-490463c5eed4" containerName="revoke-keys"
Oct 30 06:05:21.639610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.639580    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.660394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.660364    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"]
Oct 30 06:05:21.660744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.660507    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1dbaf6-openshift-default-dockercfg-vtk9c\""
Oct 30 06:05:21.778253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778215    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.778253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778255    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9zj\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-kube-api-access-tz9zj\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.778458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778312    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.778498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778461    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.778498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778491    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.778571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778518    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.778608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778566    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fd3513-4813-4402-b48c-8a5a761c45c9-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.778608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778600    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.778713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.778701    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.880671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.880597    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.880671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.880628    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9zj\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-kube-api-access-tz9zj\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.880882 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.880758    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.880882 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.880838    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.880882 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.880866    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.880902    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.880926    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fd3513-4813-4402-b48c-8a5a761c45c9-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.880955    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.881055    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.881065    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.881052    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.881218    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.881277    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.881629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.881606    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fd3513-4813-4402-b48c-8a5a761c45c9-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.883504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.883486    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.883676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.883659    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.904079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.904053    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.904673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.904650    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9zj\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-kube-api-access-tz9zj\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:21.957759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:21.957729    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:22.314387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.314362    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"]
Oct 30 06:05:22.316096 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:05:22.316062    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fd3513_4813_4402_b48c_8a5a761c45c9.slice/crio-b1f138937594abb4e4ecb4097c9230849b90bf4366434df9b619c3011ac8083d WatchSource:0}: Error finding container b1f138937594abb4e4ecb4097c9230849b90bf4366434df9b619c3011ac8083d: Status 404 returned error can't find the container with id b1f138937594abb4e4ecb4097c9230849b90bf4366434df9b619c3011ac8083d
Oct 30 06:05:22.318850 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.318819    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:22.318917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.318886    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:22.318953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.318915    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:22.468197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.467239    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:22.670106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.669155    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:22.718204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.718008    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"]
Oct 30 06:05:22.748348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.748315    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:22.763292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.762494    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"]
Oct 30 06:05:22.770464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.770377    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:05:22.770464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.770443    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:05:22.911838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.911809    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:22.912009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.911957    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:22.912009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:22.911986    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c645f\" (UniqueName: \"kubernetes.io/projected/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-kube-api-access-c645f\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:23.016620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.016534    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:23.016620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.016577    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c645f\" (UniqueName: \"kubernetes.io/projected/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-kube-api-access-c645f\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:23.016890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.016872    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:23.017675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.017648    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:23.019787 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.019765    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:23.022877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.022851    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" event={"ID":"83fd3513-4813-4402-b48c-8a5a761c45c9","Type":"ContainerStarted","Data":"e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682"}
Oct 30 06:05:23.023003 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.022942    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" event={"ID":"83fd3513-4813-4402-b48c-8a5a761c45c9","Type":"ContainerStarted","Data":"b1f138937594abb4e4ecb4097c9230849b90bf4366434df9b619c3011ac8083d"}
Oct 30 06:05:23.038631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.038610    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c645f\" (UniqueName: \"kubernetes.io/projected/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-kube-api-access-c645f\") pod \"maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:23.072660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.072591    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podStartSLOduration=2.072572829 podStartE2EDuration="2.072572829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:23.067886152 +0000 UTC m=+2037.898923370" watchObservedRunningTime="2026-10-30 06:05:23.072572829 +0000 UTC m=+2037.903610043"
Oct 30 06:05:23.102386 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.102358    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:23.416243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.416220    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"]
Oct 30 06:05:23.419700 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:05:23.419668    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6eeb83_ade3_4b51_8ed1_e240459f07e2.slice/crio-d3827ed4d7d82e00c494b63ce4cc69fad20a9b9ccf6a7dc53d28d63fa9759f77 WatchSource:0}: Error finding container d3827ed4d7d82e00c494b63ce4cc69fad20a9b9ccf6a7dc53d28d63fa9759f77: Status 404 returned error can't find the container with id d3827ed4d7d82e00c494b63ce4cc69fad20a9b9ccf6a7dc53d28d63fa9759f77
Oct 30 06:05:23.963262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.962838    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:23.963262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.962953    5250 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 06:05:23.963262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:23.962995    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 06:05:24.032170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:24.032137    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47" event={"ID":"cb6eeb83-ade3-4b51-8ed1-e240459f07e2","Type":"ContainerStarted","Data":"d3827ed4d7d82e00c494b63ce4cc69fad20a9b9ccf6a7dc53d28d63fa9759f77"}
Oct 30 06:05:24.799966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:24.799831    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:24.962206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:24.959558    5250 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 06:05:24.962206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:24.959617    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 06:05:25.043210 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:25.043109    5250 generic.go:356] "Generic (PLEG): container finished" podID="cb6eeb83-ade3-4b51-8ed1-e240459f07e2" containerID="2eac04a2693ee3230ae145f3db2bce7099de52927cfb110ace1b02d2b5687ef3" exitCode=0
Oct 30 06:05:25.043210 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:25.043149    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47" event={"ID":"cb6eeb83-ade3-4b51-8ed1-e240459f07e2","Type":"ContainerDied","Data":"2eac04a2693ee3230ae145f3db2bce7099de52927cfb110ace1b02d2b5687ef3"}
Oct 30 06:05:25.973614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:25.972603    5250 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 06:05:25.973614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:25.972677    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 06:05:26.094507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.094427    5250 generic.go:356] "Generic (PLEG): container finished" podID="b096884b-f166-45ff-9590-81d96cba7d2c" containerID="ec0ce2d400890f81b8ca279a12faadc381d8c14825a1c06e48f37061fd074a9a" exitCode=0
Oct 30 06:05:26.094790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.094761    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" event={"ID":"b096884b-f166-45ff-9590-81d96cba7d2c","Type":"ContainerDied","Data":"ec0ce2d400890f81b8ca279a12faadc381d8c14825a1c06e48f37061fd074a9a"}
Oct 30 06:05:26.140657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.140628    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 06:05:26.294952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.294911    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-credential-socket\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.295119 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.295094    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-envoy\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.295214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.295156    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-kube-api-access-jh6j7\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-kube-api-access-jh6j7\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.295271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.295215    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-socket\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.295334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.295292    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-istio-token\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-istio-token\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.295417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.295400    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-certs\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.295496 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.295481    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-data\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.295566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.295552    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b096884b-f166-45ff-9590-81d96cba7d2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096884b-f166-45ff-9590-81d96cba7d2c-istiod-ca-cert\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.295623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.295588    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b096884b-f166-45ff-9590-81d96cba7d2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096884b-f166-45ff-9590-81d96cba7d2c-istio-podinfo\") pod \"b096884b-f166-45ff-9590-81d96cba7d2c\" (UID: \"b096884b-f166-45ff-9590-81d96cba7d2c\") "
Oct 30 06:05:26.297899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.297867    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-data" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:26.298211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.298166    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b096884b-f166-45ff-9590-81d96cba7d2c-istiod-ca-cert" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:05:26.299857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.299739    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-certs" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:26.300091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.299952    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-credential-socket" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:26.300333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.300145    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-socket" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:26.303331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.303305    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b096884b-f166-45ff-9590-81d96cba7d2c-istio-podinfo" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:05:26.303952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.303918    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-envoy" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:05:26.304045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.303958    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-kube-api-access-jh6j7" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "kube-api-access-jh6j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:26.305026 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.304992    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-istio-token" pod "b096884b-f166-45ff-9590-81d96cba7d2c" (UID: "b096884b-f166-45ff-9590-81d96cba7d2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:26.338609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.338574    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.402825    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096884b-f166-45ff-9590-81d96cba7d2c-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.403150    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096884b-f166-45ff-9590-81d96cba7d2c-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.403164    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.403198    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.403213    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh6j7\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-kube-api-access-jh6j7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.403227    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.403242    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096884b-f166-45ff-9590-81d96cba7d2c-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.403258    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.403997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.403270    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096884b-f166-45ff-9590-81d96cba7d2c-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.503853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.503821    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-kube-api-access-c645f\" (UniqueName: \"kubernetes.io/projected/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-kube-api-access-c645f\") pod \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") "
Oct 30 06:05:26.504084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.503912    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-openshift-ca\") pod \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") "
Oct 30 06:05:26.504084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.504063    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-xks-ca\") pod \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\" (UID: \"cb6eeb83-ade3-4b51-8ed1-e240459f07e2\") "
Oct 30 06:05:26.504402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.504372    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-openshift-ca" pod "cb6eeb83-ade3-4b51-8ed1-e240459f07e2" (UID: "cb6eeb83-ade3-4b51-8ed1-e240459f07e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:05:26.504914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.504894    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.506777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.506751    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-xks-ca" pod "cb6eeb83-ade3-4b51-8ed1-e240459f07e2" (UID: "cb6eeb83-ade3-4b51-8ed1-e240459f07e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:05:26.506893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.506821    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-kube-api-access-c645f" pod "cb6eeb83-ade3-4b51-8ed1-e240459f07e2" (UID: "cb6eeb83-ade3-4b51-8ed1-e240459f07e2"). InnerVolumeSpecName "kube-api-access-c645f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:05:26.606724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.606686    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c645f\" (UniqueName: \"kubernetes.io/projected/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-kube-api-access-c645f\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.606724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.606721    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6eeb83-ade3-4b51-8ed1-e240459f07e2-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:05:26.731655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.730488    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:26.974491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.974453    5250 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 06:05:26.975096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:26.975065    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 06:05:27.104220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.104096    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"
Oct 30 06:05:27.104220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.104125    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb" event={"ID":"b096884b-f166-45ff-9590-81d96cba7d2c","Type":"ContainerDied","Data":"700461a81bc2a2640c7e2684c378bb76d0cd9f1ca32d5ad59a5f591bd93a1458"}
Oct 30 06:05:27.104220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.104217    5250 scope.go:122] "RemoveContainer" containerID="ec0ce2d400890f81b8ca279a12faadc381d8c14825a1c06e48f37061fd074a9a"
Oct 30 06:05:27.106288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.106090    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"
Oct 30 06:05:27.106288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.106194    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47" event={"ID":"cb6eeb83-ade3-4b51-8ed1-e240459f07e2","Type":"ContainerDied","Data":"d3827ed4d7d82e00c494b63ce4cc69fad20a9b9ccf6a7dc53d28d63fa9759f77"}
Oct 30 06:05:27.106288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.106238    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3827ed4d7d82e00c494b63ce4cc69fad20a9b9ccf6a7dc53d28d63fa9759f77"
Oct 30 06:05:27.184401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.184367    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"]
Oct 30 06:05:27.188586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.188561    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-zphkb"]
Oct 30 06:05:27.296865 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.296829    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" path="/var/lib/kubelet/pods/b096884b-f166-45ff-9590-81d96cba7d2c/volumes"
Oct 30 06:05:27.959578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.959541    5250 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 06:05:27.960565 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:27.960531    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 06:05:28.505986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.505952    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"]
Oct 30 06:05:28.506826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.506808    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy"
Oct 30 06:05:28.506875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.506828    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy"
Oct 30 06:05:28.506875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.506857    5250 cpu_manager.go:414] "containerMap: removing container" podUID="cb6eeb83-ade3-4b51-8ed1-e240459f07e2" containerName="revoke-keys"
Oct 30 06:05:28.506875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.506862    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6eeb83-ade3-4b51-8ed1-e240459f07e2" containerName="revoke-keys"
Oct 30 06:05:28.507165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.506988    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b096884b-f166-45ff-9590-81d96cba7d2c" containerName="istio-proxy"
Oct 30 06:05:28.507165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.506999    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6eeb83-ade3-4b51-8ed1-e240459f07e2" containerName="revoke-keys"
Oct 30 06:05:28.512570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.512551    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.530234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.530206    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-482468-openshift-default-dockercfg-fctb5\""
Oct 30 06:05:28.546035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.546004    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"]
Oct 30 06:05:28.546958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.546932    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-data\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.547132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.547112    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.547251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.547194    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6nl\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-kube-api-access-ss6nl\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.547251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.547239    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-token\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.547377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.547361    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-podinfo\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.547443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.547427    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istiod-ca-cert\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.547502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.547488    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-envoy\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.547559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.547522    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-credential-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.547634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.547589    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-certs\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.645890    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650341    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-certs\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650413    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-data\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650527    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650578    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6nl\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-kube-api-access-ss6nl\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650609    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-token\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650699    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-podinfo\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650756    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istiod-ca-cert\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650817    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-envoy\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.650844    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-credential-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.651269    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-credential-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.651487    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-certs\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.651738    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-data\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.651956    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.672203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.653790    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istiod-ca-cert\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.711201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.710107    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6nl\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-kube-api-access-ss6nl\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.711201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.710446    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-envoy\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.714209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.713822    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-token\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.714209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.713933    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-podinfo\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.805899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.805807    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:28.827069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.827045    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:28.962621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.961901    5250 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 06:05:28.962621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.961970    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 06:05:28.988514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:28.988471    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:29.203215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:29.203045    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"]
Oct 30 06:05:29.205216 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:05:29.205161    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f52ebb8_6bcf_4483_b7a1_4f2233315baf.slice/crio-0412540ff4dae10c718ec19f6b6e28a61454de8f6aac95000cf80cd1167f249b WatchSource:0}: Error finding container 0412540ff4dae10c718ec19f6b6e28a61454de8f6aac95000cf80cd1167f249b: Status 404 returned error can't find the container with id 0412540ff4dae10c718ec19f6b6e28a61454de8f6aac95000cf80cd1167f249b
Oct 30 06:05:29.209370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:29.209329    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:29.209489 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:29.209415    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:29.209489 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:29.209454    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:29.978218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:29.977558    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:29.980204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:29.979367    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:29.981762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:29.981732    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:05:30.147531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:30.141525    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" event={"ID":"5f52ebb8-6bcf-4483-b7a1-4f2233315baf","Type":"ContainerStarted","Data":"2d1af394e4e45c5c8adbe9d70061c41c0d294d4b4bf4e2c881d1b54b19a6196d"}
Oct 30 06:05:30.147531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:30.141582    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" event={"ID":"5f52ebb8-6bcf-4483-b7a1-4f2233315baf","Type":"ContainerStarted","Data":"0412540ff4dae10c718ec19f6b6e28a61454de8f6aac95000cf80cd1167f249b"}
Oct 30 06:05:30.188154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:30.188103    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" podStartSLOduration=2.1880888 podStartE2EDuration="2.1880888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:30.18364924 +0000 UTC m=+2045.014686458" watchObservedRunningTime="2026-10-30 06:05:30.1880888 +0000 UTC m=+2045.019126017"
Oct 30 06:05:30.776230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:30.773712    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:30.827383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:30.827356    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:30.828867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:30.828841    5250 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 06:05:30.828998 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:30.828888    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 06:05:31.827728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:31.827692    5250 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 06:05:31.828130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:31.827759    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 06:05:32.401382 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:05:32.401345    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:05:32.637639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:32.637587    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:32.828403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:32.828313    5250 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 06:05:32.828403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:32.828381    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 06:05:33.353318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:33.353270    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:33.560204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:33.555548    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:33.827636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:33.827593    5250 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 06:05:33.827856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:33.827661    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 06:05:34.428272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:34.428236    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:34.828166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:34.828076    5250 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 06:05:34.828166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:34.828144    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 06:05:35.040024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:35.039987    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:35.327407 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:35.327372    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:35.903624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:35.903579    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:35.904382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:35.904359    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:35.905246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:35.905226    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:05:36.506941 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:36.506903    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:37.823296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:37.823262    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:38.642737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.642701    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:38.643779 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.643746    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"]
Oct 30 06:05:38.649441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.649418    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.676408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.676377    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8de592-openshift-default-dockercfg-kzz2m\""
Oct 30 06:05:38.688740 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.688705    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"]
Oct 30 06:05:38.762887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.762855    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea392143-910f-4440-961a-508594fd5d6f-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.763070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.762898    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea392143-910f-4440-961a-508594fd5d6f-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.763070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.763004    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjzg\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-kube-api-access-5kjzg\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.763070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.763049    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.763218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.763173    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.763295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.763281    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.763335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.763320    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.763400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.763387    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.763454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.763448    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.864964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.864933    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea392143-910f-4440-961a-508594fd5d6f-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.865501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.864989    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjzg\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-kube-api-access-5kjzg\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.865501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865018    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.865501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865223    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.865501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865277    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.865501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865326    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.865501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865378    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.865501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865478    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.866048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865513    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.866048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865656    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea392143-910f-4440-961a-508594fd5d6f-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.866048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865705    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.866048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865776    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.866048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.865970    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.866386 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.866231    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea392143-910f-4440-961a-508594fd5d6f-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.867668 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.867641    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.867974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.867953    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea392143-910f-4440-961a-508594fd5d6f-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.887272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.887244    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.887272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.887266    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjzg\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-kube-api-access-5kjzg\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:38.986081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:38.986002    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:39.326220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:39.326173    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"]
Oct 30 06:05:39.327588 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:05:39.327555    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea392143_910f_4440_961a_508594fd5d6f.slice/crio-03fe2922129cfab99bf058ba0c883e6953cceae1715e1c40abd0514f8b08c40c WatchSource:0}: Error finding container 03fe2922129cfab99bf058ba0c883e6953cceae1715e1c40abd0514f8b08c40c: Status 404 returned error can't find the container with id 03fe2922129cfab99bf058ba0c883e6953cceae1715e1c40abd0514f8b08c40c
Oct 30 06:05:39.330706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:39.330675    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:39.330789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:39.330745    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:39.330789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:39.330775    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:05:40.249848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:40.249810    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" event={"ID":"ea392143-910f-4440-961a-508594fd5d6f","Type":"ContainerStarted","Data":"e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22"}
Oct 30 06:05:40.250309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:40.249868    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" event={"ID":"ea392143-910f-4440-961a-508594fd5d6f","Type":"ContainerStarted","Data":"03fe2922129cfab99bf058ba0c883e6953cceae1715e1c40abd0514f8b08c40c"}
Oct 30 06:05:40.313416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:40.313359    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podStartSLOduration=2.313342961 podStartE2EDuration="2.313342961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:40.301337725 +0000 UTC m=+2055.132374953" watchObservedRunningTime="2026-10-30 06:05:40.313342961 +0000 UTC m=+2055.144380178"
Oct 30 06:05:40.684681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:40.684643    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:40.986567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:40.986479    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:40.988139 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:40.988103    5250 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 06:05:40.988348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:40.988151    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 06:05:41.247327 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:41.247207    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:41.781661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:41.781625    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:41.986937 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:41.986898    5250 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 06:05:41.987120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:41.986955    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 06:05:42.298600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:42.298566    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:42.464250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:42.455661    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:42.607258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:42.605057    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:42.986671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:42.986625    5250 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 06:05:42.987228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:42.986709    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 06:05:43.987275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:43.987226    5250 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 06:05:43.987699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:43.987291    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 06:05:44.413504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:44.413471    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:44.987206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:44.986648    5250 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 06:05:44.987206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:44.986708    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 06:05:45.986778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:45.986742    5250 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 06:05:45.986980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:45.986799    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 06:05:46.362970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:46.362864    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:46.926156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:46.926095    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:46.991989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:46.991956    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:46.992469 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:46.992448    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:46.993102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:46.993082    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:05:47.137617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:47.137575    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:48.722437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:48.722396    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:48.897611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:48.897570    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:50.668434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:50.668401    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:52.648676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:52.642373    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:54.484593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:54.484554    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:56.053103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:56.053006    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:56.781498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:56.781468    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:56.978873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:56.978839    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"]
Oct 30 06:05:56.983619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:56.983590    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:56.988064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:56.988038    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0382c2-5pq47"]
Oct 30 06:05:57.175992 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:57.175936    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:57.284158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:57.284123    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6eeb83-ade3-4b51-8ed1-e240459f07e2" path="/var/lib/kubelet/pods/cb6eeb83-ade3-4b51-8ed1-e240459f07e2/volumes"
Oct 30 06:05:57.669331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:57.669297    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:05:59.171010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:05:59.170975    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:01.785655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:01.785617    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:02.454308 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:06:02.454270    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:06:04.019458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:04.019415    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:04.835777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:04.835748    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:04.991854 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:04.991819    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:05.213644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:05.213614    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:06.126341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:06.126261    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:08.099248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:08.099213    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:09.991841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:09.991810    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:11.398057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:11.398024    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:11.640775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:11.640742    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:13.532127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.532086    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:13.683019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.682988    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"]
Oct 30 06:06:13.689436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.689410    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.718661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.718632    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"]
Oct 30 06:06:13.762674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.762645    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.762837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.762698    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75ns\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-kube-api-access-x75ns\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.762837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.762769    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.762951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.762934    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.762991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.762961    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.763060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.763044    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.763115 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.763104    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.763163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.763139    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.763349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.763326    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.791694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.791629    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-080391-openshift-default-dockercfg-w5wmr\""
Oct 30 06:06:13.864936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.864906    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.864958    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75ns\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-kube-api-access-x75ns\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.864977    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865071    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865093    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865136    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865242    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865337    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865486    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865493    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865678    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865818 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865798    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.865922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865901    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.866017 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.865994    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.868043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.868019    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.868433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.868410    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.886724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.886697    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:13.886836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:13.886819    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75ns\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-kube-api-access-x75ns\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:14.007200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:14.007155    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:14.340444 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:14.340415    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"]
Oct 30 06:06:14.341153 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:06:14.341125    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf03b07e_8e6b_4b74_aa6f_2c59bc4ff5d6.slice/crio-27ee0653650b4cb5fbe2395a124688006dbdd5449ece831226f02fa57e7a1056 WatchSource:0}: Error finding container 27ee0653650b4cb5fbe2395a124688006dbdd5449ece831226f02fa57e7a1056: Status 404 returned error can't find the container with id 27ee0653650b4cb5fbe2395a124688006dbdd5449ece831226f02fa57e7a1056
Oct 30 06:06:14.344124 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:14.344088    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:06:14.344299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:14.344163    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:06:14.344299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:14.344257    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:06:14.741506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:14.741465    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" event={"ID":"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6","Type":"ContainerStarted","Data":"d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da"}
Oct 30 06:06:14.742004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:14.741528    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" event={"ID":"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6","Type":"ContainerStarted","Data":"27ee0653650b4cb5fbe2395a124688006dbdd5449ece831226f02fa57e7a1056"}
Oct 30 06:06:14.803712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:14.803656    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" podStartSLOduration=1.803641074 podStartE2EDuration="1.803641074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:14.792169858 +0000 UTC m=+2089.623207074" watchObservedRunningTime="2026-10-30 06:06:14.803641074 +0000 UTC m=+2089.634678290"
Oct 30 06:06:15.007926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:15.007845    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:15.009499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:15.009474    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 06:06:15.009578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:15.009532    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 06:06:15.320531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:15.315969    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:16.008201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:16.008151    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 06:06:16.008705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:16.008254    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 06:06:16.220409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:16.220374    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"]
Oct 30 06:06:16.251452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:16.251420    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"]
Oct 30 06:06:16.406542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:16.406513    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:16.552399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:16.552315    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:17.008374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:17.008335    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 06:06:17.009073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:17.009044    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 06:06:17.566646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:17.566610    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:18.008442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:18.008404    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 06:06:18.008851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:18.008459    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 06:06:19.007797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:19.007758    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 06:06:19.007996 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:19.007811    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 06:06:19.514299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:19.514261    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:20.007505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:20.007464    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 06:06:20.007700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:20.007525    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 06:06:21.012610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.012580    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:21.013104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.012886    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:21.013637 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.013621    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:06:21.220909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.220850    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg" podUID="266ffdf0-d47f-4ffa-a023-3d939555934d" containerName="payload-processing" containerID="cri-o://3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb" gracePeriod=25
Oct 30 06:06:21.252592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.252558    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb" podUID="08c56799-0c32-4d43-97ae-0912b9ae86ec" containerName="payload-pre-processing" containerID="cri-o://31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa" gracePeriod=25
Oct 30 06:06:21.365085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.365004    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:21.545500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.545475    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:06:21.673244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.673150    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266ffdf0-d47f-4ffa-a023-3d939555934d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ffdf0-d47f-4ffa-a023-3d939555934d-plugins-config-volume\") pod \"266ffdf0-d47f-4ffa-a023-3d939555934d\" (UID: \"266ffdf0-d47f-4ffa-a023-3d939555934d\") "
Oct 30 06:06:21.673434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.673327    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266ffdf0-d47f-4ffa-a023-3d939555934d-kube-api-access-h5pp9\" (UniqueName: \"kubernetes.io/projected/266ffdf0-d47f-4ffa-a023-3d939555934d-kube-api-access-h5pp9\") pod \"266ffdf0-d47f-4ffa-a023-3d939555934d\" (UID: \"266ffdf0-d47f-4ffa-a023-3d939555934d\") "
Oct 30 06:06:21.673545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.673523    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266ffdf0-d47f-4ffa-a023-3d939555934d-plugins-config-volume" pod "266ffdf0-d47f-4ffa-a023-3d939555934d" (UID: "266ffdf0-d47f-4ffa-a023-3d939555934d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:06:21.674056 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.673970    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ffdf0-d47f-4ffa-a023-3d939555934d-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:21.675626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.675603    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266ffdf0-d47f-4ffa-a023-3d939555934d-kube-api-access-h5pp9" pod "266ffdf0-d47f-4ffa-a023-3d939555934d" (UID: "266ffdf0-d47f-4ffa-a023-3d939555934d"). InnerVolumeSpecName "kube-api-access-h5pp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:06:21.775472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.775442    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5pp9\" (UniqueName: \"kubernetes.io/projected/266ffdf0-d47f-4ffa-a023-3d939555934d-kube-api-access-h5pp9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:21.826277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.826243    5250 generic.go:356] "Generic (PLEG): container finished" podID="266ffdf0-d47f-4ffa-a023-3d939555934d" containerID="3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb" exitCode=0
Oct 30 06:06:21.826449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.826316    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"
Oct 30 06:06:21.826449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.826337    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg" event={"ID":"266ffdf0-d47f-4ffa-a023-3d939555934d","Type":"ContainerDied","Data":"3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb"}
Oct 30 06:06:21.826449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.826399    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg" event={"ID":"266ffdf0-d47f-4ffa-a023-3d939555934d","Type":"ContainerDied","Data":"f5a24fcf30a44d9fb5f1223a8baa5e7a8d5e7e030577608e3101063ddd0f6dbc"}
Oct 30 06:06:21.826449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.826423    5250 scope.go:122] "RemoveContainer" containerID="3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb"
Oct 30 06:06:21.842380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.842357    5250 scope.go:122] "RemoveContainer" containerID="3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb"
Oct 30 06:06:21.842752 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:06:21.842727    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb\": container with ID starting with 3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb not found: ID does not exist" containerID="3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb"
Oct 30 06:06:21.842839 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.842766    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb"} err="failed to get container status \"3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb\": rpc error: code = NotFound desc = could not find container \"3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb\": container with ID starting with 3cdd15e32e7b0b90df50df6bca3812925d1ddc35acbe8b8fd53df7e3f98c6fdb not found: ID does not exist"
Oct 30 06:06:21.877566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.877534    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"]
Oct 30 06:06:21.882343 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:21.882319    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765de9-7675966898-l2jwg"]
Oct 30 06:06:23.283598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:23.283563    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266ffdf0-d47f-4ffa-a023-3d939555934d" path="/var/lib/kubelet/pods/266ffdf0-d47f-4ffa-a023-3d939555934d/volumes"
Oct 30 06:06:23.301620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:23.300445    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:25.048234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:25.048199    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:25.774729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:25.774701    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:06:25.870662 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:25.870634    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:06:26.706519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:26.706470    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:27.829359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:27.829317    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb" podUID="08c56799-0c32-4d43-97ae-0912b9ae86ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:06:28.607663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:28.607591    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:30.281919 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:30.281878    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:31.996009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:31.995965    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:32.375585 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:06:32.375504    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:06:32.808671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:32.803147    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:33.323031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.322995    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"]
Oct 30 06:06:33.324522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.324492    5250 cpu_manager.go:414] "containerMap: removing container" podUID="266ffdf0-d47f-4ffa-a023-3d939555934d" containerName="payload-processing"
Oct 30 06:06:33.324522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.324518    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266ffdf0-d47f-4ffa-a023-3d939555934d" containerName="payload-processing"
Oct 30 06:06:33.324722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.324680    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="266ffdf0-d47f-4ffa-a023-3d939555934d" containerName="payload-processing"
Oct 30 06:06:33.332781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.332753    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.350309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.350279    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:06:33.350520 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.350316    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:06:33.367572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.366738    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"]
Oct 30 06:06:33.369793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.369575    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.369793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.369671    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.370058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.369814    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9r7\" (UniqueName: \"kubernetes.io/projected/599f341c-e18f-427c-9e76-22fce7270c85-kube-api-access-hs9r7\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.472656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.472624    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.472802 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.472693    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.472802 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.472771    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9r7\" (UniqueName: \"kubernetes.io/projected/599f341c-e18f-427c-9e76-22fce7270c85-kube-api-access-hs9r7\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.473495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.473422    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.475430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.475408    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.494968 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.494940    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9r7\" (UniqueName: \"kubernetes.io/projected/599f341c-e18f-427c-9e76-22fce7270c85-kube-api-access-hs9r7\") pod \"maas-api-revoke-sub-0ab564fc-5srjh\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.675032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.674998    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:33.947848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:33.947761    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:34.008521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:34.008458    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"]
Oct 30 06:06:34.011951 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:06:34.011922    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599f341c_e18f_427c_9e76_22fce7270c85.slice/crio-edb5195b78dcf97a024446f4566e964249e7951d8ff408b52938637650f14eac WatchSource:0}: Error finding container edb5195b78dcf97a024446f4566e964249e7951d8ff408b52938637650f14eac: Status 404 returned error can't find the container with id edb5195b78dcf97a024446f4566e964249e7951d8ff408b52938637650f14eac
Oct 30 06:06:34.974325 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:34.974290    5250 generic.go:356] "Generic (PLEG): container finished" podID="599f341c-e18f-427c-9e76-22fce7270c85" containerID="ffbffec34f985c665374147b4927cc643846f179466ee2cb6e10ea01160b6dd0" exitCode=0
Oct 30 06:06:34.974725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:34.974380    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh" event={"ID":"599f341c-e18f-427c-9e76-22fce7270c85","Type":"ContainerDied","Data":"ffbffec34f985c665374147b4927cc643846f179466ee2cb6e10ea01160b6dd0"}
Oct 30 06:06:34.974725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:34.974444    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh" event={"ID":"599f341c-e18f-427c-9e76-22fce7270c85","Type":"ContainerStarted","Data":"edb5195b78dcf97a024446f4566e964249e7951d8ff408b52938637650f14eac"}
Oct 30 06:06:35.169691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:35.169611    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:35.451806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:35.451726    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:35.703593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:35.703519    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:36.163902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.163879    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:36.215154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.215121    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-openshift-ca\") pod \"599f341c-e18f-427c-9e76-22fce7270c85\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") "
Oct 30 06:06:36.215338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.215164    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-xks-ca\") pod \"599f341c-e18f-427c-9e76-22fce7270c85\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") "
Oct 30 06:06:36.215454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.215424    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599f341c-e18f-427c-9e76-22fce7270c85-kube-api-access-hs9r7\" (UniqueName: \"kubernetes.io/projected/599f341c-e18f-427c-9e76-22fce7270c85-kube-api-access-hs9r7\") pod \"599f341c-e18f-427c-9e76-22fce7270c85\" (UID: \"599f341c-e18f-427c-9e76-22fce7270c85\") "
Oct 30 06:06:36.215588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.215549    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-openshift-ca" pod "599f341c-e18f-427c-9e76-22fce7270c85" (UID: "599f341c-e18f-427c-9e76-22fce7270c85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:06:36.216025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.216008    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:36.217507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.217484    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-xks-ca" pod "599f341c-e18f-427c-9e76-22fce7270c85" (UID: "599f341c-e18f-427c-9e76-22fce7270c85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:06:36.217719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.217701    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599f341c-e18f-427c-9e76-22fce7270c85-kube-api-access-hs9r7" pod "599f341c-e18f-427c-9e76-22fce7270c85" (UID: "599f341c-e18f-427c-9e76-22fce7270c85"). InnerVolumeSpecName "kube-api-access-hs9r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:06:36.317562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.317529    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599f341c-e18f-427c-9e76-22fce7270c85-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:36.317562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.317563    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs9r7\" (UniqueName: \"kubernetes.io/projected/599f341c-e18f-427c-9e76-22fce7270c85-kube-api-access-hs9r7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:36.992604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.992570    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"
Oct 30 06:06:36.992808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.992606    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh" event={"ID":"599f341c-e18f-427c-9e76-22fce7270c85","Type":"ContainerDied","Data":"edb5195b78dcf97a024446f4566e964249e7951d8ff408b52938637650f14eac"}
Oct 30 06:06:36.992808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:36.992683    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edb5195b78dcf97a024446f4566e964249e7951d8ff408b52938637650f14eac"
Oct 30 06:06:37.443675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:37.443633    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:37.601045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:37.601006    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:37.829042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:37.828960    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb" podUID="08c56799-0c32-4d43-97ae-0912b9ae86ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:06:37.878714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:37.878685    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:39.856701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:39.856665    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:41.547158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:41.547130    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:06:41.644677 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:41.644356    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:41.653447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:41.652056    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c56799-0c32-4d43-97ae-0912b9ae86ec-kube-api-access-9cjnx\" (UniqueName: \"kubernetes.io/projected/08c56799-0c32-4d43-97ae-0912b9ae86ec-kube-api-access-9cjnx\") pod \"08c56799-0c32-4d43-97ae-0912b9ae86ec\" (UID: \"08c56799-0c32-4d43-97ae-0912b9ae86ec\") "
Oct 30 06:06:41.653447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:41.652224    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c56799-0c32-4d43-97ae-0912b9ae86ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c56799-0c32-4d43-97ae-0912b9ae86ec-plugins-config-volume\") pod \"08c56799-0c32-4d43-97ae-0912b9ae86ec\" (UID: \"08c56799-0c32-4d43-97ae-0912b9ae86ec\") "
Oct 30 06:06:41.653870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:41.653841    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c56799-0c32-4d43-97ae-0912b9ae86ec-plugins-config-volume" pod "08c56799-0c32-4d43-97ae-0912b9ae86ec" (UID: "08c56799-0c32-4d43-97ae-0912b9ae86ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:06:41.656842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:41.656815    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c56799-0c32-4d43-97ae-0912b9ae86ec-kube-api-access-9cjnx" pod "08c56799-0c32-4d43-97ae-0912b9ae86ec" (UID: "08c56799-0c32-4d43-97ae-0912b9ae86ec"). InnerVolumeSpecName "kube-api-access-9cjnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:06:41.755339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:41.755305    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c56799-0c32-4d43-97ae-0912b9ae86ec-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:41.755339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:41.755335    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cjnx\" (UniqueName: \"kubernetes.io/projected/08c56799-0c32-4d43-97ae-0912b9ae86ec-kube-api-access-9cjnx\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:42.055333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.055297    5250 generic.go:356] "Generic (PLEG): container finished" podID="08c56799-0c32-4d43-97ae-0912b9ae86ec" containerID="31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa" exitCode=0
Oct 30 06:06:42.055509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.055339    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"
Oct 30 06:06:42.055509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.055377    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb" event={"ID":"08c56799-0c32-4d43-97ae-0912b9ae86ec","Type":"ContainerDied","Data":"31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa"}
Oct 30 06:06:42.055509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.055434    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb" event={"ID":"08c56799-0c32-4d43-97ae-0912b9ae86ec","Type":"ContainerDied","Data":"e63874d321891d5f057f0595e25150ea5f753f679cc7388f6f4728ad9532caaf"}
Oct 30 06:06:42.055509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.055456    5250 scope.go:122] "RemoveContainer" containerID="31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa"
Oct 30 06:06:42.068401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.068383    5250 scope.go:122] "RemoveContainer" containerID="31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa"
Oct 30 06:06:42.068681 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:06:42.068660    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa\": container with ID starting with 31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa not found: ID does not exist" containerID="31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa"
Oct 30 06:06:42.068766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.068691    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa"} err="failed to get container status \"31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa\": rpc error: code = NotFound desc = could not find container \"31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa\": container with ID starting with 31862792daad3079a84f51f296c800622a27f0d6fbe465b4358a7778870304fa not found: ID does not exist"
Oct 30 06:06:42.104722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.104696    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"]
Oct 30 06:06:42.114274 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.114251    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765de9-84954dfbc-q5bdb"]
Oct 30 06:06:42.201012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.200978    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"]
Oct 30 06:06:42.201862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.201840    5250 cpu_manager.go:414] "containerMap: removing container" podUID="599f341c-e18f-427c-9e76-22fce7270c85" containerName="revoke-keys"
Oct 30 06:06:42.201862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.201862    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599f341c-e18f-427c-9e76-22fce7270c85" containerName="revoke-keys"
Oct 30 06:06:42.202027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.201894    5250 cpu_manager.go:414] "containerMap: removing container" podUID="08c56799-0c32-4d43-97ae-0912b9ae86ec" containerName="payload-pre-processing"
Oct 30 06:06:42.202027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.201899    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c56799-0c32-4d43-97ae-0912b9ae86ec" containerName="payload-pre-processing"
Oct 30 06:06:42.202095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.202027    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c56799-0c32-4d43-97ae-0912b9ae86ec" containerName="payload-pre-processing"
Oct 30 06:06:42.202095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.202043    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="599f341c-e18f-427c-9e76-22fce7270c85" containerName="revoke-keys"
Oct 30 06:06:42.207356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.207337    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.225319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.223476    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:06:42.225319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.223536    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:06:42.229362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.229327    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"]
Oct 30 06:06:42.363784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.363703    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64942300-c73a-492a-a57a-c48930e12985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.363939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.363825    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64942300-c73a-492a-a57a-c48930e12985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.363939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.363894    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8qr\" (UniqueName: \"kubernetes.io/projected/64942300-c73a-492a-a57a-c48930e12985-kube-api-access-js8qr\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.466054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.466016    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64942300-c73a-492a-a57a-c48930e12985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.466246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.466083    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64942300-c73a-492a-a57a-c48930e12985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.466246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.466124    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8qr\" (UniqueName: \"kubernetes.io/projected/64942300-c73a-492a-a57a-c48930e12985-kube-api-access-js8qr\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.466891 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.466871    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64942300-c73a-492a-a57a-c48930e12985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.468727 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.468704    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64942300-c73a-492a-a57a-c48930e12985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.487589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.487565    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8qr\" (UniqueName: \"kubernetes.io/projected/64942300-c73a-492a-a57a-c48930e12985-kube-api-access-js8qr\") pod \"maas-api-revoke-sub-bdd36098-nhmrb\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.537033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.537005    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:42.851890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:42.851861    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"]
Oct 30 06:06:42.853689 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:06:42.853655    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64942300_c73a_492a_a57a_c48930e12985.slice/crio-89a511f61455d2783be476b9b31e0efab950432a4aba77aa63f76953c9680e4a WatchSource:0}: Error finding container 89a511f61455d2783be476b9b31e0efab950432a4aba77aa63f76953c9680e4a: Status 404 returned error can't find the container with id 89a511f61455d2783be476b9b31e0efab950432a4aba77aa63f76953c9680e4a
Oct 30 06:06:43.067070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:43.067035    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb" event={"ID":"64942300-c73a-492a-a57a-c48930e12985","Type":"ContainerStarted","Data":"89a511f61455d2783be476b9b31e0efab950432a4aba77aa63f76953c9680e4a"}
Oct 30 06:06:43.284078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:43.284044    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c56799-0c32-4d43-97ae-0912b9ae86ec" path="/var/lib/kubelet/pods/08c56799-0c32-4d43-97ae-0912b9ae86ec/volumes"
Oct 30 06:06:43.405252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:43.403027    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:44.079838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:44.079739    5250 generic.go:356] "Generic (PLEG): container finished" podID="64942300-c73a-492a-a57a-c48930e12985" containerID="2f66addbdbf83385ca0428b691b0013e97b947ef6647ff577316afc0789ad84e" exitCode=0
Oct 30 06:06:44.080256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:44.079814    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb" event={"ID":"64942300-c73a-492a-a57a-c48930e12985","Type":"ContainerDied","Data":"2f66addbdbf83385ca0428b691b0013e97b947ef6647ff577316afc0789ad84e"}
Oct 30 06:06:45.125721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.125690    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:45.303956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.303930    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:45.322962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.322939    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64942300-c73a-492a-a57a-c48930e12985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64942300-c73a-492a-a57a-c48930e12985-maas-api-openshift-ca\") pod \"64942300-c73a-492a-a57a-c48930e12985\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") "
Oct 30 06:06:45.323121 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.323016    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64942300-c73a-492a-a57a-c48930e12985-kube-api-access-js8qr\" (UniqueName: \"kubernetes.io/projected/64942300-c73a-492a-a57a-c48930e12985-kube-api-access-js8qr\") pod \"64942300-c73a-492a-a57a-c48930e12985\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") "
Oct 30 06:06:45.323272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.323146    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64942300-c73a-492a-a57a-c48930e12985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64942300-c73a-492a-a57a-c48930e12985-maas-api-xks-ca\") pod \"64942300-c73a-492a-a57a-c48930e12985\" (UID: \"64942300-c73a-492a-a57a-c48930e12985\") "
Oct 30 06:06:45.323396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.323371    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64942300-c73a-492a-a57a-c48930e12985-maas-api-openshift-ca" pod "64942300-c73a-492a-a57a-c48930e12985" (UID: "64942300-c73a-492a-a57a-c48930e12985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:06:45.323931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.323910    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64942300-c73a-492a-a57a-c48930e12985-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:45.325361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.325336    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64942300-c73a-492a-a57a-c48930e12985-kube-api-access-js8qr" pod "64942300-c73a-492a-a57a-c48930e12985" (UID: "64942300-c73a-492a-a57a-c48930e12985"). InnerVolumeSpecName "kube-api-access-js8qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:06:45.325455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.325384    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64942300-c73a-492a-a57a-c48930e12985-maas-api-xks-ca" pod "64942300-c73a-492a-a57a-c48930e12985" (UID: "64942300-c73a-492a-a57a-c48930e12985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:06:45.425830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.425797    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8qr\" (UniqueName: \"kubernetes.io/projected/64942300-c73a-492a-a57a-c48930e12985-kube-api-access-js8qr\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:45.425830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:45.425828    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64942300-c73a-492a-a57a-c48930e12985-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:46.097452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:46.097426    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"
Oct 30 06:06:46.097626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:46.097420    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb" event={"ID":"64942300-c73a-492a-a57a-c48930e12985","Type":"ContainerDied","Data":"89a511f61455d2783be476b9b31e0efab950432a4aba77aa63f76953c9680e4a"}
Oct 30 06:06:46.097626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:46.097547    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89a511f61455d2783be476b9b31e0efab950432a4aba77aa63f76953c9680e4a"
Oct 30 06:06:46.261430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:46.261400    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:46.422840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:46.422806    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:46.852556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:46.852462    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:48.703528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:48.703492    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:50.598823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:50.598790    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:52.371744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:52.371705    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:54.101055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:54.101023    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:55.521909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.521870    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:55.774089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.774000    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"]
Oct 30 06:06:55.775361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.775339    5250 cpu_manager.go:414] "containerMap: removing container" podUID="64942300-c73a-492a-a57a-c48930e12985" containerName="revoke-keys"
Oct 30 06:06:55.775436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.775365    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64942300-c73a-492a-a57a-c48930e12985" containerName="revoke-keys"
Oct 30 06:06:55.775567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.775553    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="64942300-c73a-492a-a57a-c48930e12985" containerName="revoke-keys"
Oct 30 06:06:55.781515 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.781478    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:55.798699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.798673    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"]
Oct 30 06:06:55.802805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.802781    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:06:55.802933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.802878    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:06:55.900525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.900491    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjpp\" (UniqueName: \"kubernetes.io/projected/18d75f2c-dca5-45e8-b543-022d9e6c00c6-kube-api-access-krjpp\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:55.900525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.900526    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:55.900764 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:55.900679    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:56.002057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.002023    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjpp\" (UniqueName: \"kubernetes.io/projected/18d75f2c-dca5-45e8-b543-022d9e6c00c6-kube-api-access-krjpp\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:56.002293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.002072    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:56.002293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.002198    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:56.002784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.002760    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:56.011068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.011028    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:56.026500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.026436    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjpp\" (UniqueName: \"kubernetes.io/projected/18d75f2c-dca5-45e8-b543-022d9e6c00c6-kube-api-access-krjpp\") pod \"maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:56.083604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.083568    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:56.102971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.102932    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:56.476427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:56.476400    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"]
Oct 30 06:06:56.477900 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:06:56.477877    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d75f2c_dca5_45e8_b543_022d9e6c00c6.slice/crio-3e645d20ffebf70ac9c5eab04784187ffb799daecacecc4d412c002dc8d7c444 WatchSource:0}: Error finding container 3e645d20ffebf70ac9c5eab04784187ffb799daecacecc4d412c002dc8d7c444: Status 404 returned error can't find the container with id 3e645d20ffebf70ac9c5eab04784187ffb799daecacecc4d412c002dc8d7c444
Oct 30 06:06:57.212545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:57.212509    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5" event={"ID":"18d75f2c-dca5-45e8-b543-022d9e6c00c6","Type":"ContainerStarted","Data":"ae31a2abab75847580fbf7c162a4d900007fa4777f259c9f00a4b8f153eb0c47"}
Oct 30 06:06:57.212948 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:57.212563    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5" event={"ID":"18d75f2c-dca5-45e8-b543-022d9e6c00c6","Type":"ContainerStarted","Data":"3e645d20ffebf70ac9c5eab04784187ffb799daecacecc4d412c002dc8d7c444"}
Oct 30 06:06:57.253416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:57.253367    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5" podStartSLOduration=2.2533524050000002 podStartE2EDuration="2.253352405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:57.248789032 +0000 UTC m=+2132.079826248" watchObservedRunningTime="2026-10-30 06:06:57.253352405 +0000 UTC m=+2132.084389622"
Oct 30 06:06:58.212326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:58.212293    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:06:58.236257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:58.236218    5250 generic.go:356] "Generic (PLEG): container finished" podID="18d75f2c-dca5-45e8-b543-022d9e6c00c6" containerID="ae31a2abab75847580fbf7c162a4d900007fa4777f259c9f00a4b8f153eb0c47" exitCode=0
Oct 30 06:06:58.237085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:58.236485    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5" event={"ID":"18d75f2c-dca5-45e8-b543-022d9e6c00c6","Type":"ContainerDied","Data":"ae31a2abab75847580fbf7c162a4d900007fa4777f259c9f00a4b8f153eb0c47"}
Oct 30 06:06:59.419280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.419256    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:06:59.461790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.461660    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d75f2c-dca5-45e8-b543-022d9e6c00c6-kube-api-access-krjpp\" (UniqueName: \"kubernetes.io/projected/18d75f2c-dca5-45e8-b543-022d9e6c00c6-kube-api-access-krjpp\") pod \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") "
Oct 30 06:06:59.462220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.462193    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-xks-ca\") pod \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") "
Oct 30 06:06:59.462521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.462491    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-openshift-ca\") pod \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\" (UID: \"18d75f2c-dca5-45e8-b543-022d9e6c00c6\") "
Oct 30 06:06:59.463079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.463039    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-openshift-ca" pod "18d75f2c-dca5-45e8-b543-022d9e6c00c6" (UID: "18d75f2c-dca5-45e8-b543-022d9e6c00c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:06:59.463687 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.463664    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:59.465140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.464826    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d75f2c-dca5-45e8-b543-022d9e6c00c6-kube-api-access-krjpp" pod "18d75f2c-dca5-45e8-b543-022d9e6c00c6" (UID: "18d75f2c-dca5-45e8-b543-022d9e6c00c6"). InnerVolumeSpecName "kube-api-access-krjpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:06:59.465391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.465366    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-xks-ca" pod "18d75f2c-dca5-45e8-b543-022d9e6c00c6" (UID: "18d75f2c-dca5-45e8-b543-022d9e6c00c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:06:59.565623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.565552    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krjpp\" (UniqueName: \"kubernetes.io/projected/18d75f2c-dca5-45e8-b543-022d9e6c00c6-kube-api-access-krjpp\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:59.565623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.565579    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d75f2c-dca5-45e8-b543-022d9e6c00c6-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:06:59.992640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:06:59.992551    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:00.254459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:00.254360    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5" event={"ID":"18d75f2c-dca5-45e8-b543-022d9e6c00c6","Type":"ContainerDied","Data":"3e645d20ffebf70ac9c5eab04784187ffb799daecacecc4d412c002dc8d7c444"}
Oct 30 06:07:00.254459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:00.254390    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"
Oct 30 06:07:00.254459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:00.254435    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e645d20ffebf70ac9c5eab04784187ffb799daecacecc4d412c002dc8d7c444"
Oct 30 06:07:01.804129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:01.804094    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:02.375719 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:07:02.375684    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:07:03.270621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:03.270586    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:04.093761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:04.093730    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:04.236643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:04.236601    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:04.376317 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:04.376027    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:04.751839 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:04.751796    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:06.378471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:06.378433    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:06.548050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:06.547202    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:08.175603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:08.175575    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:09.963370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:09.963338    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:10.939097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:10.939036    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:11.091993 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:11.091959    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:11.298495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:11.298417    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:12.134388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:12.130864    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:13.980698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:13.980663    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:14.514816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:14.514779    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"]
Oct 30 06:07:14.558884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:14.558852    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:15.890012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:15.889977    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:17.720219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:17.720153    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:19.386789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:19.386752    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:21.081670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:21.081627    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:22.724167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:22.724132    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:22.870388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:22.870357    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:23.091105 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.090966    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:23.403540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.403492    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"]
Oct 30 06:07:23.466613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.466525    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:23.937686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.937651    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"]
Oct 30 06:07:23.938480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.938463    5250 cpu_manager.go:414] "containerMap: removing container" podUID="18d75f2c-dca5-45e8-b543-022d9e6c00c6" containerName="revoke-keys"
Oct 30 06:07:23.938543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.938480    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d75f2c-dca5-45e8-b543-022d9e6c00c6" containerName="revoke-keys"
Oct 30 06:07:23.938646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.938636    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d75f2c-dca5-45e8-b543-022d9e6c00c6" containerName="revoke-keys"
Oct 30 06:07:23.942566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.942544    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:23.959717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.959693    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"]
Oct 30 06:07:23.976474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.976444    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:07:23.976625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:23.976483    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:07:24.007242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.007170    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.007439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.007423    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.007623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.007606    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdwj4\" (UniqueName: \"kubernetes.io/projected/06ae2a46-476b-4396-89e3-9dd3d6bffb65-kube-api-access-pdwj4\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.108828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.108791    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdwj4\" (UniqueName: \"kubernetes.io/projected/06ae2a46-476b-4396-89e3-9dd3d6bffb65-kube-api-access-pdwj4\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.109083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.108857    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.109083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.109027    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.110228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.110162    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.111712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.111686    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.131890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.131863    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdwj4\" (UniqueName: \"kubernetes.io/projected/06ae2a46-476b-4396-89e3-9dd3d6bffb65-kube-api-access-pdwj4\") pod \"maas-api-revoke-sub-df7c3554-zkckq\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.259477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.259404    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:24.574903 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.574871    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"]
Oct 30 06:07:24.575870 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:07:24.575839    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ae2a46_476b_4396_89e3_9dd3d6bffb65.slice/crio-67a888ea9abe0123fb5e21eb55184b258d86b242d07684dfe3bb8de24ad9cbff WatchSource:0}: Error finding container 67a888ea9abe0123fb5e21eb55184b258d86b242d07684dfe3bb8de24ad9cbff: Status 404 returned error can't find the container with id 67a888ea9abe0123fb5e21eb55184b258d86b242d07684dfe3bb8de24ad9cbff
Oct 30 06:07:24.816701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:24.816664    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:25.569312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:25.569279    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq" event={"ID":"06ae2a46-476b-4396-89e3-9dd3d6bffb65","Type":"ContainerStarted","Data":"9052d8cfec4d85d77235d21ba80be0ab33121bee933126df70005ff17e10faf7"}
Oct 30 06:07:25.569712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:25.569332    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq" event={"ID":"06ae2a46-476b-4396-89e3-9dd3d6bffb65","Type":"ContainerStarted","Data":"67a888ea9abe0123fb5e21eb55184b258d86b242d07684dfe3bb8de24ad9cbff"}
Oct 30 06:07:25.611282 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:25.611233    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq" podStartSLOduration=2.6112193870000002 podStartE2EDuration="2.611219387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:25.608140739 +0000 UTC m=+2160.439177968" watchObservedRunningTime="2026-10-30 06:07:25.611219387 +0000 UTC m=+2160.442256612"
Oct 30 06:07:26.576841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:26.576801    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:26.580769 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:26.580744    5250 generic.go:356] "Generic (PLEG): container finished" podID="06ae2a46-476b-4396-89e3-9dd3d6bffb65" containerID="9052d8cfec4d85d77235d21ba80be0ab33121bee933126df70005ff17e10faf7" exitCode=0
Oct 30 06:07:26.580935 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:26.580822    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq" event={"ID":"06ae2a46-476b-4396-89e3-9dd3d6bffb65","Type":"ContainerDied","Data":"9052d8cfec4d85d77235d21ba80be0ab33121bee933126df70005ff17e10faf7"}
Oct 30 06:07:27.765728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.765699    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:27.870785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.870755    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ae2a46-476b-4396-89e3-9dd3d6bffb65-kube-api-access-pdwj4\" (UniqueName: \"kubernetes.io/projected/06ae2a46-476b-4396-89e3-9dd3d6bffb65-kube-api-access-pdwj4\") pod \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") "
Oct 30 06:07:27.870990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.870968    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-openshift-ca\") pod \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") "
Oct 30 06:07:27.871223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.871118    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-xks-ca\") pod \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\" (UID: \"06ae2a46-476b-4396-89e3-9dd3d6bffb65\") "
Oct 30 06:07:27.871423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.871399    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-openshift-ca" pod "06ae2a46-476b-4396-89e3-9dd3d6bffb65" (UID: "06ae2a46-476b-4396-89e3-9dd3d6bffb65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:07:27.872067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.872046    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:27.873503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.873477    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-xks-ca" pod "06ae2a46-476b-4396-89e3-9dd3d6bffb65" (UID: "06ae2a46-476b-4396-89e3-9dd3d6bffb65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:07:27.873503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.873481    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ae2a46-476b-4396-89e3-9dd3d6bffb65-kube-api-access-pdwj4" pod "06ae2a46-476b-4396-89e3-9dd3d6bffb65" (UID: "06ae2a46-476b-4396-89e3-9dd3d6bffb65"). InnerVolumeSpecName "kube-api-access-pdwj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:07:27.973153 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.973109    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdwj4\" (UniqueName: \"kubernetes.io/projected/06ae2a46-476b-4396-89e3-9dd3d6bffb65-kube-api-access-pdwj4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:27.973153 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:27.973144    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ae2a46-476b-4396-89e3-9dd3d6bffb65-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:28.593101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:28.593062    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:28.602917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:28.602892    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"
Oct 30 06:07:28.603085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:28.602963    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq" event={"ID":"06ae2a46-476b-4396-89e3-9dd3d6bffb65","Type":"ContainerDied","Data":"67a888ea9abe0123fb5e21eb55184b258d86b242d07684dfe3bb8de24ad9cbff"}
Oct 30 06:07:28.603085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:28.603016    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a888ea9abe0123fb5e21eb55184b258d86b242d07684dfe3bb8de24ad9cbff"
Oct 30 06:07:28.728825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:28.728784    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:29.490823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.490788    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:29.549295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.549254    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth" podUID="d24adee0-b7bb-47bc-b838-f7af41449825" containerName="main" containerID="cri-o://a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8" gracePeriod=45
Oct 30 06:07:29.863592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.863567    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:07:29.928807 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.928770    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-tmp-dir\") pod \"d24adee0-b7bb-47bc-b838-f7af41449825\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") "
Oct 30 06:07:29.929021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.928900    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-home\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-home\") pod \"d24adee0-b7bb-47bc-b838-f7af41449825\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") "
Oct 30 06:07:29.929021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.928987    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24adee0-b7bb-47bc-b838-f7af41449825-tls-certs\" (UniqueName: \"kubernetes.io/secret/d24adee0-b7bb-47bc-b838-f7af41449825-tls-certs\") pod \"d24adee0-b7bb-47bc-b838-f7af41449825\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") "
Oct 30 06:07:29.929216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.929095    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-model-cache\") pod \"d24adee0-b7bb-47bc-b838-f7af41449825\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") "
Oct 30 06:07:29.929216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.929125    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24adee0-b7bb-47bc-b838-f7af41449825-kube-api-access-8v22f\" (UniqueName: \"kubernetes.io/projected/d24adee0-b7bb-47bc-b838-f7af41449825-kube-api-access-8v22f\") pod \"d24adee0-b7bb-47bc-b838-f7af41449825\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") "
Oct 30 06:07:29.929216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.929154    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-dshm\") pod \"d24adee0-b7bb-47bc-b838-f7af41449825\" (UID: \"d24adee0-b7bb-47bc-b838-f7af41449825\") "
Oct 30 06:07:29.929403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.929207    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-home" pod "d24adee0-b7bb-47bc-b838-f7af41449825" (UID: "d24adee0-b7bb-47bc-b838-f7af41449825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:29.929468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.929409    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-model-cache" pod "d24adee0-b7bb-47bc-b838-f7af41449825" (UID: "d24adee0-b7bb-47bc-b838-f7af41449825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:29.930032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.930008    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:29.930220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.930204    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:29.930355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.930334    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-tmp-dir" pod "d24adee0-b7bb-47bc-b838-f7af41449825" (UID: "d24adee0-b7bb-47bc-b838-f7af41449825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:29.934358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.934331    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24adee0-b7bb-47bc-b838-f7af41449825-kube-api-access-8v22f" pod "d24adee0-b7bb-47bc-b838-f7af41449825" (UID: "d24adee0-b7bb-47bc-b838-f7af41449825"). InnerVolumeSpecName "kube-api-access-8v22f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:07:29.940598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.940573    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24adee0-b7bb-47bc-b838-f7af41449825-tls-certs" pod "d24adee0-b7bb-47bc-b838-f7af41449825" (UID: "d24adee0-b7bb-47bc-b838-f7af41449825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:07:29.940695 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:29.940598    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-dshm" pod "d24adee0-b7bb-47bc-b838-f7af41449825" (UID: "d24adee0-b7bb-47bc-b838-f7af41449825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:30.031859 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.031828    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:30.031859 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.031856    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24adee0-b7bb-47bc-b838-f7af41449825-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:30.031859 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.031865    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v22f\" (UniqueName: \"kubernetes.io/projected/d24adee0-b7bb-47bc-b838-f7af41449825-kube-api-access-8v22f\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:30.032067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.031874    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24adee0-b7bb-47bc-b838-f7af41449825-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:30.597137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.597098    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:30.643529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.643460    5250 generic.go:356] "Generic (PLEG): container finished" podID="d24adee0-b7bb-47bc-b838-f7af41449825" containerID="a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8" exitCode=0
Oct 30 06:07:30.644051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.643531    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"
Oct 30 06:07:30.644374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.643554    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth" event={"ID":"d24adee0-b7bb-47bc-b838-f7af41449825","Type":"ContainerDied","Data":"a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8"}
Oct 30 06:07:30.644522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.644396    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth" event={"ID":"d24adee0-b7bb-47bc-b838-f7af41449825","Type":"ContainerDied","Data":"1e7730fc89a632d57be185bf3367e873ed8984953275fad941ae02fc4a1031d6"}
Oct 30 06:07:30.644522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.644419    5250 scope.go:122] "RemoveContainer" containerID="a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8"
Oct 30 06:07:30.660413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.660263    5250 scope.go:122] "RemoveContainer" containerID="a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8"
Oct 30 06:07:30.661376 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:07:30.661347    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8\": container with ID starting with a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8 not found: ID does not exist" containerID="a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8"
Oct 30 06:07:30.661482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.661386    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8"} err="failed to get container status \"a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8\": rpc error: code = NotFound desc = could not find container \"a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8\": container with ID starting with a9648c3a6695bd6cf4c98d294f81c2710df8a61e0c531e8e5ecea53cd53166a8 not found: ID does not exist"
Oct 30 06:07:30.701102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.701066    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"]
Oct 30 06:07:30.711108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:30.711069    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/auth-test-model-1b4b7a-kserve-9f8697d49-54rth"]
Oct 30 06:07:31.334205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:31.328941    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24adee0-b7bb-47bc-b838-f7af41449825" path="/var/lib/kubelet/pods/d24adee0-b7bb-47bc-b838-f7af41449825/volumes"
Oct 30 06:07:32.375231 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:07:32.375193    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:07:32.485849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:32.485816    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:35.024956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:35.024909    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:35.174606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:35.174568    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:35.325230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:35.325082    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:37.004698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.004415    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"]
Oct 30 06:07:37.010855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.010814    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:37.157503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.157337    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:37.307096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.307015    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"]
Oct 30 06:07:37.307898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.307878    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d24adee0-b7bb-47bc-b838-f7af41449825" containerName="main"
Oct 30 06:07:37.307898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.307897    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24adee0-b7bb-47bc-b838-f7af41449825" containerName="main"
Oct 30 06:07:37.308066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.307938    5250 cpu_manager.go:414] "containerMap: removing container" podUID="06ae2a46-476b-4396-89e3-9dd3d6bffb65" containerName="revoke-keys"
Oct 30 06:07:37.308066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.307943    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ae2a46-476b-4396-89e3-9dd3d6bffb65" containerName="revoke-keys"
Oct 30 06:07:37.308066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.308062    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ae2a46-476b-4396-89e3-9dd3d6bffb65" containerName="revoke-keys"
Oct 30 06:07:37.308270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.308076    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24adee0-b7bb-47bc-b838-f7af41449825" containerName="main"
Oct 30 06:07:37.313658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.313637    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.329071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.329009    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:07:37.333114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.332342    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"]
Oct 30 06:07:37.338203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.337717    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:07:37.439354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.439318    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxqpt\" (UniqueName: \"kubernetes.io/projected/e5ba6160-f995-4baa-a187-6ed272669a84-kube-api-access-rxqpt\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.439558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.439527    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.439684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.439663    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.541984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.541948    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.542229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.542161    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxqpt\" (UniqueName: \"kubernetes.io/projected/e5ba6160-f995-4baa-a187-6ed272669a84-kube-api-access-rxqpt\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.542336 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.542287    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.542784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.542754    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.545173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.545150    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.566722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.566656    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxqpt\" (UniqueName: \"kubernetes.io/projected/e5ba6160-f995-4baa-a187-6ed272669a84-kube-api-access-rxqpt\") pod \"maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.637699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.637665    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:37.957350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:37.957316    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"]
Oct 30 06:07:37.957894 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:07:37.957864    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ba6160_f995_4baa_a187_6ed272669a84.slice/crio-1e1d26a324ca15c5ae304d79dc6c268f7f856c1ce7c0ae60c717c2fb27d7b4a1 WatchSource:0}: Error finding container 1e1d26a324ca15c5ae304d79dc6c268f7f856c1ce7c0ae60c717c2fb27d7b4a1: Status 404 returned error can't find the container with id 1e1d26a324ca15c5ae304d79dc6c268f7f856c1ce7c0ae60c717c2fb27d7b4a1
Oct 30 06:07:38.433841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.433804    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr" podUID="66283eba-062d-419c-8f0a-449481d5c02b" containerName="main" containerID="cri-o://07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34" gracePeriod=45
Oct 30 06:07:38.732371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.732347    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:07:38.848571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.848478    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b" event={"ID":"e5ba6160-f995-4baa-a187-6ed272669a84","Type":"ContainerStarted","Data":"837cbf03e977b5a1ee6ca845c7efc0451648ee4748166375daee5c9bac1a2f34"}
Oct 30 06:07:38.848571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.848538    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b" event={"ID":"e5ba6160-f995-4baa-a187-6ed272669a84","Type":"ContainerStarted","Data":"1e1d26a324ca15c5ae304d79dc6c268f7f856c1ce7c0ae60c717c2fb27d7b4a1"}
Oct 30 06:07:38.850295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.850217    5250 generic.go:356] "Generic (PLEG): container finished" podID="66283eba-062d-419c-8f0a-449481d5c02b" containerID="07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34" exitCode=0
Oct 30 06:07:38.850453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.850383    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"
Oct 30 06:07:38.850503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.850389    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr" event={"ID":"66283eba-062d-419c-8f0a-449481d5c02b","Type":"ContainerDied","Data":"07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34"}
Oct 30 06:07:38.850550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.850516    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr" event={"ID":"66283eba-062d-419c-8f0a-449481d5c02b","Type":"ContainerDied","Data":"e57d2e688a83bd78c5bdc634d0bcf98ffe9b3ccc1364bb5b356f32d64755fe86"}
Oct 30 06:07:38.850550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.850544    5250 scope.go:122] "RemoveContainer" containerID="07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34"
Oct 30 06:07:38.861215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.861154    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-model-cache\") pod \"66283eba-062d-419c-8f0a-449481d5c02b\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") "
Oct 30 06:07:38.861342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.861236    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66283eba-062d-419c-8f0a-449481d5c02b-kube-api-access-p5ksb\" (UniqueName: \"kubernetes.io/projected/66283eba-062d-419c-8f0a-449481d5c02b-kube-api-access-p5ksb\") pod \"66283eba-062d-419c-8f0a-449481d5c02b\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") "
Oct 30 06:07:38.861436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.861363    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-dshm\") pod \"66283eba-062d-419c-8f0a-449481d5c02b\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") "
Oct 30 06:07:38.861436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.861429    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66283eba-062d-419c-8f0a-449481d5c02b-tls-certs\" (UniqueName: \"kubernetes.io/secret/66283eba-062d-419c-8f0a-449481d5c02b-tls-certs\") pod \"66283eba-062d-419c-8f0a-449481d5c02b\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") "
Oct 30 06:07:38.861582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.861463    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-model-cache" pod "66283eba-062d-419c-8f0a-449481d5c02b" (UID: "66283eba-062d-419c-8f0a-449481d5c02b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:38.861582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.861564    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-tmp-dir\") pod \"66283eba-062d-419c-8f0a-449481d5c02b\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") "
Oct 30 06:07:38.861755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.861602    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-home\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-home\") pod \"66283eba-062d-419c-8f0a-449481d5c02b\" (UID: \"66283eba-062d-419c-8f0a-449481d5c02b\") "
Oct 30 06:07:38.862361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.862340    5250 scope.go:122] "RemoveContainer" containerID="07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34"
Oct 30 06:07:38.862463 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.862357    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:38.862825 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:07:38.862799    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34\": container with ID starting with 07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34 not found: ID does not exist" containerID="07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34"
Oct 30 06:07:38.862911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.862843    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34"} err="failed to get container status \"07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34\": rpc error: code = NotFound desc = could not find container \"07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34\": container with ID starting with 07649cb3dd9ce4ee03f9b7cc9c7c4f3b037475d60a845ee0cc92476b10b1ca34 not found: ID does not exist"
Oct 30 06:07:38.862911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.862807    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-tmp-dir" pod "66283eba-062d-419c-8f0a-449481d5c02b" (UID: "66283eba-062d-419c-8f0a-449481d5c02b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:38.862911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.862848    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-home" pod "66283eba-062d-419c-8f0a-449481d5c02b" (UID: "66283eba-062d-419c-8f0a-449481d5c02b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:38.863810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.863791    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-dshm" pod "66283eba-062d-419c-8f0a-449481d5c02b" (UID: "66283eba-062d-419c-8f0a-449481d5c02b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:38.863938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.863922    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66283eba-062d-419c-8f0a-449481d5c02b-kube-api-access-p5ksb" pod "66283eba-062d-419c-8f0a-449481d5c02b" (UID: "66283eba-062d-419c-8f0a-449481d5c02b"). InnerVolumeSpecName "kube-api-access-p5ksb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:07:38.864438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.864419    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66283eba-062d-419c-8f0a-449481d5c02b-tls-certs" pod "66283eba-062d-419c-8f0a-449481d5c02b" (UID: "66283eba-062d-419c-8f0a-449481d5c02b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:07:38.886679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.886628    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b" podStartSLOduration=1.8866157970000001 podStartE2EDuration="1.886615797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:38.885216896 +0000 UTC m=+2173.716254110" watchObservedRunningTime="2026-10-30 06:07:38.886615797 +0000 UTC m=+2173.717653041"
Oct 30 06:07:38.963594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.963566    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ksb\" (UniqueName: \"kubernetes.io/projected/66283eba-062d-419c-8f0a-449481d5c02b-kube-api-access-p5ksb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:38.963594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.963592    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:38.963775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.963608    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66283eba-062d-419c-8f0a-449481d5c02b-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:38.963775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.963622    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:38.963775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:38.963630    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66283eba-062d-419c-8f0a-449481d5c02b-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:39.194008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:39.193959    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"]
Oct 30 06:07:39.204067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:39.204044    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1dbaf6/test-model-1dbaf6-kserve-65fcb77d9f-n4ckr"]
Oct 30 06:07:39.284223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:39.284169    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66283eba-062d-419c-8f0a-449481d5c02b" path="/var/lib/kubelet/pods/66283eba-062d-419c-8f0a-449481d5c02b/volumes"
Oct 30 06:07:39.356933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:39.356901    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:39.861997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:39.861910    5250 generic.go:356] "Generic (PLEG): container finished" podID="e5ba6160-f995-4baa-a187-6ed272669a84" containerID="837cbf03e977b5a1ee6ca845c7efc0451648ee4748166375daee5c9bac1a2f34" exitCode=0
Oct 30 06:07:39.861997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:39.861965    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b" event={"ID":"e5ba6160-f995-4baa-a187-6ed272669a84","Type":"ContainerDied","Data":"837cbf03e977b5a1ee6ca845c7efc0451648ee4748166375daee5c9bac1a2f34"}
Oct 30 06:07:41.068133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.068108    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:41.096947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.096914    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-xks-ca\") pod \"e5ba6160-f995-4baa-a187-6ed272669a84\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") "
Oct 30 06:07:41.097091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.096983    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ba6160-f995-4baa-a187-6ed272669a84-kube-api-access-rxqpt\" (UniqueName: \"kubernetes.io/projected/e5ba6160-f995-4baa-a187-6ed272669a84-kube-api-access-rxqpt\") pod \"e5ba6160-f995-4baa-a187-6ed272669a84\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") "
Oct 30 06:07:41.097091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.097017    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-openshift-ca\") pod \"e5ba6160-f995-4baa-a187-6ed272669a84\" (UID: \"e5ba6160-f995-4baa-a187-6ed272669a84\") "
Oct 30 06:07:41.097441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.097417    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-openshift-ca" pod "e5ba6160-f995-4baa-a187-6ed272669a84" (UID: "e5ba6160-f995-4baa-a187-6ed272669a84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:07:41.097773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.097715    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:41.099298 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.099274    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ba6160-f995-4baa-a187-6ed272669a84-kube-api-access-rxqpt" pod "e5ba6160-f995-4baa-a187-6ed272669a84" (UID: "e5ba6160-f995-4baa-a187-6ed272669a84"). InnerVolumeSpecName "kube-api-access-rxqpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:07:41.099401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.099318    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-xks-ca" pod "e5ba6160-f995-4baa-a187-6ed272669a84" (UID: "e5ba6160-f995-4baa-a187-6ed272669a84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:07:41.108780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.108744    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:41.200509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.200476    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba6160-f995-4baa-a187-6ed272669a84-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:41.200509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.200508    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxqpt\" (UniqueName: \"kubernetes.io/projected/e5ba6160-f995-4baa-a187-6ed272669a84-kube-api-access-rxqpt\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:41.889934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.889901    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"
Oct 30 06:07:41.890126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.889975    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b" event={"ID":"e5ba6160-f995-4baa-a187-6ed272669a84","Type":"ContainerDied","Data":"1e1d26a324ca15c5ae304d79dc6c268f7f856c1ce7c0ae60c717c2fb27d7b4a1"}
Oct 30 06:07:41.890126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.890034    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e1d26a324ca15c5ae304d79dc6c268f7f856c1ce7c0ae60c717c2fb27d7b4a1"
Oct 30 06:07:41.957794 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:41.957757    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:43.146484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:43.146445    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:44.812922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:44.812867    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:46.538980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:46.538948    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:47.714960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:47.714927    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"]
Oct 30 06:07:47.742930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:47.742899    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"]
Oct 30 06:07:47.871240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:47.871204    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:48.016424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:48.016333    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:48.825982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:48.825944    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:50.621773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:50.621739    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:52.030936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.030895    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f" podUID="fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" containerName="main" containerID="cri-o://9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03" gracePeriod=45
Oct 30 06:07:52.327060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.327028    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:07:52.482665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.482634    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-dshm\") pod \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") "
Oct 30 06:07:52.482837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.482757    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-home\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-home\") pod \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") "
Oct 30 06:07:52.482892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.482877    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tls-certs\") pod \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") "
Oct 30 06:07:52.482945 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.482930    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-model-cache\") pod \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") "
Oct 30 06:07:52.482983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.482970    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tmp-dir\") pod \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") "
Oct 30 06:07:52.483051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.483023    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-home" pod "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" (UID: "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:52.483107 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.483088    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-kube-api-access-gq4pw\" (UniqueName: \"kubernetes.io/projected/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-kube-api-access-gq4pw\") pod \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\" (UID: \"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e\") "
Oct 30 06:07:52.483243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.483220    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-model-cache" pod "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" (UID: "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:52.483334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.483261    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tmp-dir" pod "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" (UID: "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:52.483777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.483759    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:52.483875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.483778    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:52.483875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.483787    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:52.485213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.485173    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-dshm" pod "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" (UID: "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:07:52.485365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.485347    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tls-certs" pod "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" (UID: "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:07:52.485440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.485417    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-kube-api-access-gq4pw" pod "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" (UID: "fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e"). InnerVolumeSpecName "kube-api-access-gq4pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:07:52.555759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.555671    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:52.586306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.586262    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:52.586306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.586297    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq4pw\" (UniqueName: \"kubernetes.io/projected/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-kube-api-access-gq4pw\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:52.586306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.586311    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:07:52.715967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.715928    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc" podUID="dd834ca2-0b67-496c-9cb0-bab44ac5e884" containerName="payload-processing" containerID="cri-o://0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4" gracePeriod=25
Oct 30 06:07:52.743867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:52.743804    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5" podUID="7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" containerName="payload-pre-processing" containerID="cri-o://e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff" gracePeriod=25
Oct 30 06:07:53.006243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.006212    5250 generic.go:356] "Generic (PLEG): container finished" podID="fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" containerID="9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03" exitCode=0
Oct 30 06:07:53.006437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.006261    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"
Oct 30 06:07:53.006437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.006276    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f" event={"ID":"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e","Type":"ContainerDied","Data":"9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03"}
Oct 30 06:07:53.006437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.006305    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f" event={"ID":"fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e","Type":"ContainerDied","Data":"136e4a3742e4eaa66fc3aab861798fa55d97d3a3295ad1bfce074d45aaa7319c"}
Oct 30 06:07:53.006437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.006325    5250 scope.go:122] "RemoveContainer" containerID="9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03"
Oct 30 06:07:53.019743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.019707    5250 scope.go:122] "RemoveContainer" containerID="9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03"
Oct 30 06:07:53.020106 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:07:53.020080    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03\": container with ID starting with 9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03 not found: ID does not exist" containerID="9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03"
Oct 30 06:07:53.020254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.020114    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03"} err="failed to get container status \"9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03\": rpc error: code = NotFound desc = could not find container \"9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03\": container with ID starting with 9c92198bf6d7b9c051868de6a2f2e72888590521ade356cd9bd236e7cf602d03 not found: ID does not exist"
Oct 30 06:07:53.055446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.055415    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"]
Oct 30 06:07:53.065674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.065650    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-38cfad/test-model-38cfad-kserve-68d7685f65-46k7f"]
Oct 30 06:07:53.283632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:53.283558    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" path="/var/lib/kubelet/pods/fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e/volumes"
Oct 30 06:07:54.067492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:54.067456    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:54.892165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:54.892122    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:55.030098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:55.030062    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:55.055320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:55.055293    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"]
Oct 30 06:07:55.078380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:55.078343    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-086c60-bvkb5"]
Oct 30 06:07:55.285556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:55.285483    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d75f2c-dca5-45e8-b543-022d9e6c00c6" path="/var/lib/kubelet/pods/18d75f2c-dca5-45e8-b543-022d9e6c00c6/volumes"
Oct 30 06:07:55.341804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:55.341767    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:55.621193 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:55.621074    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:57.538105 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:57.538058    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:58.371902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.371868    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"]
Oct 30 06:07:58.372720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372702    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e5ba6160-f995-4baa-a187-6ed272669a84" containerName="revoke-keys"
Oct 30 06:07:58.372793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372722    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ba6160-f995-4baa-a187-6ed272669a84" containerName="revoke-keys"
Oct 30 06:07:58.372793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372736    5250 cpu_manager.go:414] "containerMap: removing container" podUID="66283eba-062d-419c-8f0a-449481d5c02b" containerName="main"
Oct 30 06:07:58.372793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372741    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66283eba-062d-419c-8f0a-449481d5c02b" containerName="main"
Oct 30 06:07:58.372793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372760    5250 cpu_manager.go:414] "containerMap: removing container" podUID="fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" containerName="main"
Oct 30 06:07:58.372793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372765    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" containerName="main"
Oct 30 06:07:58.372946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372885    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ba6160-f995-4baa-a187-6ed272669a84" containerName="revoke-keys"
Oct 30 06:07:58.372946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372894    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="66283eba-062d-419c-8f0a-449481d5c02b" containerName="main"
Oct 30 06:07:58.372946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.372902    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcec2480-ff5a-4368-ac2f-aeaf7f1c7b0e" containerName="main"
Oct 30 06:07:58.378156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.378136    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.402244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.402217    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"]
Oct 30 06:07:58.407427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.407400    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-080391\"/\"openshift-service-ca.crt\""
Oct 30 06:07:58.407678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.407657    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-080391\"/\"kube-root-ca.crt\""
Oct 30 06:07:58.407792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.407773    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-080391\"/\"default-dockercfg-d72bm\""
Oct 30 06:07:58.407866 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.407854    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-080391\"/\"auth-test-model-080391-kserve-self-signed-certs\""
Oct 30 06:07:58.462924 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.462893    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:07:58.473782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.473755    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"]
Oct 30 06:07:58.478544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.478527    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.501611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.501566    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:07:58.501777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.501639    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:07:58.501886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.501864    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"]
Oct 30 06:07:58.508636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.508606    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgq2\" (UniqueName: \"kubernetes.io/projected/4db95a07-1966-4eff-93c8-c88baec91b3f-kube-api-access-7qgq2\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.508891 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.508866    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdhv\" (UniqueName: \"kubernetes.io/projected/6e678012-1dd4-48c5-b76d-d325bc7779b2-kube-api-access-5kdhv\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.509098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.509078    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e678012-1dd4-48c5-b76d-d325bc7779b2-tls-certs\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.509300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.509282    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-home\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.509448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.509431    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-model-cache\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.509670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.509653    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-tmp-dir\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.509849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.509831    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.509997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.509980    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-dshm\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.510161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.510144    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.611319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611288    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-tmp-dir\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611330    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611365    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-dshm\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611431    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611553    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgq2\" (UniqueName: \"kubernetes.io/projected/4db95a07-1966-4eff-93c8-c88baec91b3f-kube-api-access-7qgq2\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611609    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdhv\" (UniqueName: \"kubernetes.io/projected/6e678012-1dd4-48c5-b76d-d325bc7779b2-kube-api-access-5kdhv\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611696    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e678012-1dd4-48c5-b76d-d325bc7779b2-tls-certs\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611740    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-tmp-dir\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611774    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-home\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.611857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.611810    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-model-cache\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.612636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.612255    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-model-cache\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.612636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.612305    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.612636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.612398    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-home\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.614334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.614290    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-dshm\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.615097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.615057    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.629344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.616442    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e678012-1dd4-48c5-b76d-d325bc7779b2-tls-certs\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.680207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.676813    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgq2\" (UniqueName: \"kubernetes.io/projected/4db95a07-1966-4eff-93c8-c88baec91b3f-kube-api-access-7qgq2\") pod \"maas-api-revoke-sub-0f0de9ac-2nxzn\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.752338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.751770    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdhv\" (UniqueName: \"kubernetes.io/projected/6e678012-1dd4-48c5-b76d-d325bc7779b2-kube-api-access-5kdhv\") pod \"auth-test-model-080391-kserve-fd748b7f7-ngvqk\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") " pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:58.819885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.819459    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:07:58.994121 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:58.994090    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:07:59.151843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:59.151723    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"]
Oct 30 06:07:59.314500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:59.314469    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"]
Oct 30 06:07:59.314914 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:07:59.314888    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e678012_1dd4_48c5_b76d_d325bc7779b2.slice/crio-e2a7a6c918f646cb7e8d5b340dbf08f7749bcc9a31d8d85e7273e64d6369d859 WatchSource:0}: Error finding container e2a7a6c918f646cb7e8d5b340dbf08f7749bcc9a31d8d85e7273e64d6369d859: Status 404 returned error can't find the container with id e2a7a6c918f646cb7e8d5b340dbf08f7749bcc9a31d8d85e7273e64d6369d859
Oct 30 06:07:59.444251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:07:59.439950    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:00.085390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.085340    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk" event={"ID":"6e678012-1dd4-48c5-b76d-d325bc7779b2","Type":"ContainerStarted","Data":"7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f"}
Oct 30 06:08:00.085390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.085396    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk" event={"ID":"6e678012-1dd4-48c5-b76d-d325bc7779b2","Type":"ContainerStarted","Data":"e2a7a6c918f646cb7e8d5b340dbf08f7749bcc9a31d8d85e7273e64d6369d859"}
Oct 30 06:08:00.087487 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.087458    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn" event={"ID":"4db95a07-1966-4eff-93c8-c88baec91b3f","Type":"ContainerStarted","Data":"4ceb7b93146b4a0038974240990584686f4b46cba4beacf7df63d1fc29b41661"}
Oct 30 06:08:00.087617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.087507    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn" event={"ID":"4db95a07-1966-4eff-93c8-c88baec91b3f","Type":"ContainerStarted","Data":"e94eff774d523318a7c60ca0d52f85ba992df3ab8afbd3f3ef55f8e35e7dc590"}
Oct 30 06:08:00.129048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.128964    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk" podStartSLOduration=2.128944958 podStartE2EDuration="2.128944958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:00.126684082 +0000 UTC m=+2194.957721310" watchObservedRunningTime="2026-10-30 06:08:00.128944958 +0000 UTC m=+2194.959982179"
Oct 30 06:08:00.140804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.140769    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5" podUID="7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:00.140977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.140782    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc" podUID="dd834ca2-0b67-496c-9cb0-bab44ac5e884" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:00.164269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.164202    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn" podStartSLOduration=2.164163651 podStartE2EDuration="2.164163651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:00.162688112 +0000 UTC m=+2194.993725330" watchObservedRunningTime="2026-10-30 06:08:00.164163651 +0000 UTC m=+2194.995200869"
Oct 30 06:08:00.994753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:00.994719    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:08:01.098163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:01.098129    5250 generic.go:356] "Generic (PLEG): container finished" podID="4db95a07-1966-4eff-93c8-c88baec91b3f" containerID="4ceb7b93146b4a0038974240990584686f4b46cba4beacf7df63d1fc29b41661" exitCode=0
Oct 30 06:08:01.098600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:01.098236    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn" event={"ID":"4db95a07-1966-4eff-93c8-c88baec91b3f","Type":"ContainerDied","Data":"4ceb7b93146b4a0038974240990584686f4b46cba4beacf7df63d1fc29b41661"}
Oct 30 06:08:01.329720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:01.329609    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:01.785418 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:01.785383    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:01.926242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:01.926203    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:02.106419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.106334    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"]
Oct 30 06:08:02.106855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.106581    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy" containerID="cri-o://389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044" gracePeriod=30
Oct 30 06:08:02.116735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.116076    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"]
Oct 30 06:08:02.116735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.116404    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy" containerID="cri-o://f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23" gracePeriod=30
Oct 30 06:08:02.181017 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.180985    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:02.466490 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:02.466451    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:08:02.530780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.530755    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:08:02.634672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.634636    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"]
Oct 30 06:08:02.635468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.635451    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4db95a07-1966-4eff-93c8-c88baec91b3f" containerName="revoke-keys"
Oct 30 06:08:02.635534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.635469    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db95a07-1966-4eff-93c8-c88baec91b3f" containerName="revoke-keys"
Oct 30 06:08:02.635597 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.635589    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db95a07-1966-4eff-93c8-c88baec91b3f" containerName="revoke-keys"
Oct 30 06:08:02.639351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.639330    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.657943 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.657910    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"]
Oct 30 06:08:02.661070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.661044    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db95a07-1966-4eff-93c8-c88baec91b3f-kube-api-access-7qgq2\" (UniqueName: \"kubernetes.io/projected/4db95a07-1966-4eff-93c8-c88baec91b3f-kube-api-access-7qgq2\") pod \"4db95a07-1966-4eff-93c8-c88baec91b3f\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") "
Oct 30 06:08:02.661253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.661228    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-xks-ca\") pod \"4db95a07-1966-4eff-93c8-c88baec91b3f\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") "
Oct 30 06:08:02.661380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.661362    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-openshift-ca\") pod \"4db95a07-1966-4eff-93c8-c88baec91b3f\" (UID: \"4db95a07-1966-4eff-93c8-c88baec91b3f\") "
Oct 30 06:08:02.661873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.661848    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-openshift-ca" pod "4db95a07-1966-4eff-93c8-c88baec91b3f" (UID: "4db95a07-1966-4eff-93c8-c88baec91b3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:02.664323 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.662703    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:02.670558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.670533    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-xks-ca" pod "4db95a07-1966-4eff-93c8-c88baec91b3f" (UID: "4db95a07-1966-4eff-93c8-c88baec91b3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:02.676452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.676421    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db95a07-1966-4eff-93c8-c88baec91b3f-kube-api-access-7qgq2" pod "4db95a07-1966-4eff-93c8-c88baec91b3f" (UID: "4db95a07-1966-4eff-93c8-c88baec91b3f"). InnerVolumeSpecName "kube-api-access-7qgq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:02.764507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.764427    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.764652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.764548    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxjk\" (UniqueName: \"kubernetes.io/projected/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-kube-api-access-rbxjk\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.764724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.764701    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.764903 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.764886    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qgq2\" (UniqueName: \"kubernetes.io/projected/4db95a07-1966-4eff-93c8-c88baec91b3f-kube-api-access-7qgq2\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:02.764979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.764910    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db95a07-1966-4eff-93c8-c88baec91b3f-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:02.866325 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.866289    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.866476 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.866467    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.866545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.866513    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxjk\" (UniqueName: \"kubernetes.io/projected/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-kube-api-access-rbxjk\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.867087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.867059    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.868887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.868866    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.898501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.898470    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxjk\" (UniqueName: \"kubernetes.io/projected/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-kube-api-access-rbxjk\") pod \"maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:02.960685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:02.960652    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:03.141650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.141617    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn" event={"ID":"4db95a07-1966-4eff-93c8-c88baec91b3f","Type":"ContainerDied","Data":"e94eff774d523318a7c60ca0d52f85ba992df3ab8afbd3f3ef55f8e35e7dc590"}
Oct 30 06:08:03.142640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.141667    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e94eff774d523318a7c60ca0d52f85ba992df3ab8afbd3f3ef55f8e35e7dc590"
Oct 30 06:08:03.142640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.141691    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"
Oct 30 06:08:03.222775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.222753    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:03.275264 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:08:03.274241    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d29b7a_2253_49d8_bd07_2f30c1ae5a54.slice/crio-66561734c60437aad364e42292d0e97047a6953cbd9e6ae5f631a1fee5002424 WatchSource:0}: Error finding container 66561734c60437aad364e42292d0e97047a6953cbd9e6ae5f631a1fee5002424: Status 404 returned error can't find the container with id 66561734c60437aad364e42292d0e97047a6953cbd9e6ae5f631a1fee5002424
Oct 30 06:08:03.312173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.312140    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"]
Oct 30 06:08:03.535102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.535028    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"]
Oct 30 06:08:03.535271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.535230    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r" podUID="e9b151f1-9236-4fdf-bc13-2d670c450e5a" containerName="discovery" containerID="cri-o://bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c" gracePeriod=30
Oct 30 06:08:03.852402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.852376    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 06:08:03.879408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.879377    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-kube-api-access-vp5cw\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-kube-api-access-vp5cw\") pod \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") "
Oct 30 06:08:03.879579 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.879443    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-dns-cert\") pod \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") "
Oct 30 06:08:03.879579 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.879539    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-kubeconfig\") pod \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") "
Oct 30 06:08:03.879708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.879590    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-cacerts\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-cacerts\") pod \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") "
Oct 30 06:08:03.879774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.879716    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-ca-configmap\") pod \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") "
Oct 30 06:08:03.879842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.879792    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-token\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-token\") pod \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") "
Oct 30 06:08:03.879842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.879831    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b151f1-9236-4fdf-bc13-2d670c450e5a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b151f1-9236-4fdf-bc13-2d670c450e5a-local-certs\") pod \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\" (UID: \"e9b151f1-9236-4fdf-bc13-2d670c450e5a\") "
Oct 30 06:08:03.880735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.880131    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-ca-configmap" pod "e9b151f1-9236-4fdf-bc13-2d670c450e5a" (UID: "e9b151f1-9236-4fdf-bc13-2d670c450e5a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:03.880735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.880700    5250 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-ca-configmap\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:03.882417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.882382    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-kubeconfig" pod "e9b151f1-9236-4fdf-bc13-2d670c450e5a" (UID: "e9b151f1-9236-4fdf-bc13-2d670c450e5a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:03.882546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.882441    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-dns-cert" pod "e9b151f1-9236-4fdf-bc13-2d670c450e5a" (UID: "e9b151f1-9236-4fdf-bc13-2d670c450e5a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:03.882546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.882486    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-kube-api-access-vp5cw" pod "e9b151f1-9236-4fdf-bc13-2d670c450e5a" (UID: "e9b151f1-9236-4fdf-bc13-2d670c450e5a"). InnerVolumeSpecName "kube-api-access-vp5cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:03.882688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.882669    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-cacerts" pod "e9b151f1-9236-4fdf-bc13-2d670c450e5a" (UID: "e9b151f1-9236-4fdf-bc13-2d670c450e5a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:03.882927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.882904    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b151f1-9236-4fdf-bc13-2d670c450e5a-local-certs" pod "e9b151f1-9236-4fdf-bc13-2d670c450e5a" (UID: "e9b151f1-9236-4fdf-bc13-2d670c450e5a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:03.883029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.883014    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-token" pod "e9b151f1-9236-4fdf-bc13-2d670c450e5a" (UID: "e9b151f1-9236-4fdf-bc13-2d670c450e5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:03.982473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.982435    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp5cw\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-kube-api-access-vp5cw\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:03.982473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.982472    5250 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-csr-dns-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:03.982681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.982488    5250 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-kubeconfig\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:03.982681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.982502    5250 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9b151f1-9236-4fdf-bc13-2d670c450e5a-cacerts\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:03.982681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.982515    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b151f1-9236-4fdf-bc13-2d670c450e5a-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:03.982681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:03.982529    5250 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b151f1-9236-4fdf-bc13-2d670c450e5a-local-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:04.153049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.153012    5250 generic.go:356] "Generic (PLEG): container finished" podID="e9b151f1-9236-4fdf-bc13-2d670c450e5a" containerID="bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c" exitCode=0
Oct 30 06:08:04.153522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.153064    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"
Oct 30 06:08:04.153522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.153091    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r" event={"ID":"e9b151f1-9236-4fdf-bc13-2d670c450e5a","Type":"ContainerDied","Data":"bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c"}
Oct 30 06:08:04.153522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.153152    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r" event={"ID":"e9b151f1-9236-4fdf-bc13-2d670c450e5a","Type":"ContainerDied","Data":"fdc0ac41eeb690b17b1dd7627d64ec0068e208c617cdfddc92396b002541f2ef"}
Oct 30 06:08:04.153522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.153216    5250 scope.go:122] "RemoveContainer" containerID="bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c"
Oct 30 06:08:04.155081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.155056    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z" event={"ID":"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54","Type":"ContainerStarted","Data":"aca082e8d6aa4f05ac5efe585b21b390533612545a76dd61db7b435d406d3031"}
Oct 30 06:08:04.155267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.155108    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z" event={"ID":"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54","Type":"ContainerStarted","Data":"66561734c60437aad364e42292d0e97047a6953cbd9e6ae5f631a1fee5002424"}
Oct 30 06:08:04.170023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.170005    5250 scope.go:122] "RemoveContainer" containerID="bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c"
Oct 30 06:08:04.170344 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:04.170317    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c\": container with ID starting with bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c not found: ID does not exist" containerID="bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c"
Oct 30 06:08:04.170435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.170352    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c"} err="failed to get container status \"bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c\": rpc error: code = NotFound desc = could not find container \"bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c\": container with ID starting with bf3337ac0ec5fbc1c8b7a2ae871621ef9536d5d24a121e051537eec3b62a993c not found: ID does not exist"
Oct 30 06:08:04.205027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.204984    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z" podStartSLOduration=2.204971117 podStartE2EDuration="2.204971117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:04.202473422 +0000 UTC m=+2199.033510663" watchObservedRunningTime="2026-10-30 06:08:04.204971117 +0000 UTC m=+2199.036008333"
Oct 30 06:08:04.238405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.238379    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"]
Oct 30 06:08:04.242410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.242385    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lp26r"]
Oct 30 06:08:04.996929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:04.996898    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:05.166138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:05.166103    5250 generic.go:356] "Generic (PLEG): container finished" podID="e5d29b7a-2253-49d8-bd07-2f30c1ae5a54" containerID="aca082e8d6aa4f05ac5efe585b21b390533612545a76dd61db7b435d406d3031" exitCode=0
Oct 30 06:08:05.166566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:05.166156    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z" event={"ID":"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54","Type":"ContainerDied","Data":"aca082e8d6aa4f05ac5efe585b21b390533612545a76dd61db7b435d406d3031"}
Oct 30 06:08:05.284796 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:05.284722    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b151f1-9236-4fdf-bc13-2d670c450e5a" path="/var/lib/kubelet/pods/e9b151f1-9236-4fdf-bc13-2d670c450e5a/volumes"
Oct 30 06:08:06.053145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.053109    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"]
Oct 30 06:08:06.053399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.053375    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy" containerID="cri-o://aa20b2223e0ff59efc9bc28f9fa9e7b0ff63ead7c54738ef7215e3287a4758da" gracePeriod=30
Oct 30 06:08:06.362764 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.362737    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:06.419122 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.419091    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-openshift-ca\") pod \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") "
Oct 30 06:08:06.419349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.419133    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-kube-api-access-rbxjk\" (UniqueName: \"kubernetes.io/projected/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-kube-api-access-rbxjk\") pod \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") "
Oct 30 06:08:06.419349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.419197    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-xks-ca\") pod \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\" (UID: \"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54\") "
Oct 30 06:08:06.419557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.419525    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-openshift-ca" pod "e5d29b7a-2253-49d8-bd07-2f30c1ae5a54" (UID: "e5d29b7a-2253-49d8-bd07-2f30c1ae5a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:06.420319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.420271    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:06.421629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.421600    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-xks-ca" pod "e5d29b7a-2253-49d8-bd07-2f30c1ae5a54" (UID: "e5d29b7a-2253-49d8-bd07-2f30c1ae5a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:06.422055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.422022    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-kube-api-access-rbxjk" pod "e5d29b7a-2253-49d8-bd07-2f30c1ae5a54" (UID: "e5d29b7a-2253-49d8-bd07-2f30c1ae5a54"). InnerVolumeSpecName "kube-api-access-rbxjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:06.521566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.521529    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxjk\" (UniqueName: \"kubernetes.io/projected/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-kube-api-access-rbxjk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:06.521566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:06.521562    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.000360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.000325    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:07.185342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.185316    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"
Oct 30 06:08:07.185514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.185308    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z" event={"ID":"e5d29b7a-2253-49d8-bd07-2f30c1ae5a54","Type":"ContainerDied","Data":"66561734c60437aad364e42292d0e97047a6953cbd9e6ae5f631a1fee5002424"}
Oct 30 06:08:07.185514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.185461    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66561734c60437aad364e42292d0e97047a6953cbd9e6ae5f631a1fee5002424"
Oct 30 06:08:07.436710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.436684    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:08:07.486593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.486561    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:08:07.536238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536204    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-istio-token\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-istio-token\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.536447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536263    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-envoy\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.536447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536308    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-kube-api-access-cgpfk\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-kube-api-access-cgpfk\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.536719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536690    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-certs\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.536802 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536741    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-credential-socket\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.536862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536846    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-envoy\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.536923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536911    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-data\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.537046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536946    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-certs\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.537046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.536989    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-data\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.537046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.537018    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istiod-ca-cert\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.537474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.537441    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-credential-socket" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.537562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.537519    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-data" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.537685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.537666    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-certs" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.537792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.537773    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-data" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.537924 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.537901    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-credential-socket\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.538210 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.538095    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istiod-ca-cert" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:07.538288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.538117    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-credential-socket" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.538288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.538236    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e746240-c132-4275-b437-c7f9e65ae25c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e746240-c132-4275-b437-c7f9e65ae25c-istiod-ca-cert\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.538480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.538438    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-certs" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.538628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.538607    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e746240-c132-4275-b437-c7f9e65ae25c-istiod-ca-cert" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:07.538714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.538697    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-socket\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.539193 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.539156    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-socket" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.539317 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.539297    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-socket\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.539390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.539349    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e746240-c132-4275-b437-c7f9e65ae25c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e746240-c132-4275-b437-c7f9e65ae25c-istio-podinfo\") pod \"8e746240-c132-4275-b437-c7f9e65ae25c\" (UID: \"8e746240-c132-4275-b437-c7f9e65ae25c\") "
Oct 30 06:08:07.539688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.539658    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-socket" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.539819 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.539784    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-token\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-token\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.540241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.540218    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-kube-api-access-vjrg2\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-kube-api-access-vjrg2\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.540637 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.540616    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-podinfo\") pod \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\" (UID: \"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e\") "
Oct 30 06:08:07.541945 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.541917    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.541951    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.541968    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.541982    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.541997    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.542131    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e746240-c132-4275-b437-c7f9e65ae25c-istio-podinfo" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:08:07.542316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.542214    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.542229    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.542244    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.542270    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e746240-c132-4275-b437-c7f9e65ae25c-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.542284    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.542666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.542639    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-envoy" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.542847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.542826    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-kube-api-access-cgpfk" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "kube-api-access-cgpfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:07.543150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.543128    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-istio-token" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:07.543479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.543454    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-kube-api-access-vjrg2" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "kube-api-access-vjrg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:07.544111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.544085    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-token" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:07.545098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.545040    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-envoy" pod "8e746240-c132-4275-b437-c7f9e65ae25c" (UID: "8e746240-c132-4275-b437-c7f9e65ae25c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:07.545443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.545418    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-podinfo" pod "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" (UID: "73dd30bf-0a1e-49a6-83e1-273e9bb3c58e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:08:07.644238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.644194    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e746240-c132-4275-b437-c7f9e65ae25c-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.644238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.644235    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e746240-c132-4275-b437-c7f9e65ae25c-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.644511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.644249    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.644511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.644264    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjrg2\" (UniqueName: \"kubernetes.io/projected/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-kube-api-access-vjrg2\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.644511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.644287    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.644511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.644296    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.644511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.644304    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:07.644511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:07.644313    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgpfk\" (UniqueName: \"kubernetes.io/projected/8e746240-c132-4275-b437-c7f9e65ae25c-kube-api-access-cgpfk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:08.230208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.219523    5250 generic.go:356] "Generic (PLEG): container finished" podID="8e746240-c132-4275-b437-c7f9e65ae25c" containerID="389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044" exitCode=0
Oct 30 06:08:08.230208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.219807    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" event={"ID":"8e746240-c132-4275-b437-c7f9e65ae25c","Type":"ContainerDied","Data":"389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044"}
Oct 30 06:08:08.230208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.219861    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp" event={"ID":"8e746240-c132-4275-b437-c7f9e65ae25c","Type":"ContainerDied","Data":"ec0cb7159f9ee51ff5584eeec8c052f93fff2ff4ac4c91eef9be1ca1ddb34d42"}
Oct 30 06:08:08.230208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.219888    5250 scope.go:122] "RemoveContainer" containerID="389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044"
Oct 30 06:08:08.230208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.220036    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"
Oct 30 06:08:08.247203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.233452    5250 generic.go:356] "Generic (PLEG): container finished" podID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerID="f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23" exitCode=0
Oct 30 06:08:08.247203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.233615    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" event={"ID":"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e","Type":"ContainerDied","Data":"f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23"}
Oct 30 06:08:08.247203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.233673    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49" event={"ID":"73dd30bf-0a1e-49a6-83e1-273e9bb3c58e","Type":"ContainerDied","Data":"10e5c96551474b05c83eb7d0dfb753536ff673ff6bb2d9bc2ab9e7517b1af8cd"}
Oct 30 06:08:08.247203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.233761    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"
Oct 30 06:08:08.318776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.318676    5250 scope.go:122] "RemoveContainer" containerID="389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044"
Oct 30 06:08:08.321982 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:08.321544    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044\": container with ID starting with 389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044 not found: ID does not exist" containerID="389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044"
Oct 30 06:08:08.321982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.321583    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044"} err="failed to get container status \"389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044\": rpc error: code = NotFound desc = could not find container \"389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044\": container with ID starting with 389739f1781a61d7f8b65bb38b0beda6e0f0bc14b4a0e7dab1284f4a0f143044 not found: ID does not exist"
Oct 30 06:08:08.321982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.321609    5250 scope.go:122] "RemoveContainer" containerID="f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23"
Oct 30 06:08:08.344251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.343746    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"]
Oct 30 06:08:08.370306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.366262    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-djhkp"]
Oct 30 06:08:08.393202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.386531    5250 scope.go:122] "RemoveContainer" containerID="f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23"
Oct 30 06:08:08.399220 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:08.396661    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23\": container with ID starting with f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23 not found: ID does not exist" containerID="f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23"
Oct 30 06:08:08.399220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.396706    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23"} err="failed to get container status \"f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23\": rpc error: code = NotFound desc = could not find container \"f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23\": container with ID starting with f17003fc0b49eb74eff6d4163ed8898c397ceb7c828ed2430dbf1675681aff23 not found: ID does not exist"
Oct 30 06:08:08.460222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.460164    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"]
Oct 30 06:08:08.481436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.481283    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-stc49"]
Oct 30 06:08:08.998203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:08.996592    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:08:09.006202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:09.002942    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:09.136014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:09.135984    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:08:09.255449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:09.255421    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:08:09.283391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:09.283352    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" path="/var/lib/kubelet/pods/73dd30bf-0a1e-49a6-83e1-273e9bb3c58e/volumes"
Oct 30 06:08:09.283987 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:09.283949    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" path="/var/lib/kubelet/pods/8e746240-c132-4275-b437-c7f9e65ae25c/volumes"
Oct 30 06:08:10.140169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.140123    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5" podUID="7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:10.140805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.140414    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc" podUID="dd834ca2-0b67-496c-9cb0-bab44ac5e884" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:10.386419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.386383    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"]
Oct 30 06:08:10.387693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387663    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e9b151f1-9236-4fdf-bc13-2d670c450e5a" containerName="discovery"
Oct 30 06:08:10.387693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387693    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b151f1-9236-4fdf-bc13-2d670c450e5a" containerName="discovery"
Oct 30 06:08:10.387868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387730    5250 cpu_manager.go:414] "containerMap: removing container" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy"
Oct 30 06:08:10.387868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387738    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy"
Oct 30 06:08:10.387868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387757    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy"
Oct 30 06:08:10.387868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387764    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy"
Oct 30 06:08:10.387868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387804    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e5d29b7a-2253-49d8-bd07-2f30c1ae5a54" containerName="revoke-keys"
Oct 30 06:08:10.387868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387811    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d29b7a-2253-49d8-bd07-2f30c1ae5a54" containerName="revoke-keys"
Oct 30 06:08:10.388059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.387991    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e746240-c132-4275-b437-c7f9e65ae25c" containerName="istio-proxy"
Oct 30 06:08:10.388059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.388010    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d29b7a-2253-49d8-bd07-2f30c1ae5a54" containerName="revoke-keys"
Oct 30 06:08:10.388059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.388023    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b151f1-9236-4fdf-bc13-2d670c450e5a" containerName="discovery"
Oct 30 06:08:10.388059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.388037    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="73dd30bf-0a1e-49a6-83e1-273e9bb3c58e" containerName="istio-proxy"
Oct 30 06:08:10.395647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.395589    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.409868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.409840    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"]
Oct 30 06:08:10.413792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.413763    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:08:10.414558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.414259    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:08:10.547440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.547399    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ncj\" (UniqueName: \"kubernetes.io/projected/8ad85965-824b-4e18-8cb7-3df26685c0ff-kube-api-access-q4ncj\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.547587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.547534    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.547658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.547647    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.626042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.625981    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:10.649658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.649595    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ncj\" (UniqueName: \"kubernetes.io/projected/8ad85965-824b-4e18-8cb7-3df26685c0ff-kube-api-access-q4ncj\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.649781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.649662    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.649781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.649735    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.650374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.650350    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.652260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.652234    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.671336 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.671315    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ncj\" (UniqueName: \"kubernetes.io/projected/8ad85965-824b-4e18-8cb7-3df26685c0ff-kube-api-access-q4ncj\") pod \"maas-api-revoke-sub-68a34f05-68zqd\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.740127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.740102    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:10.784532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:10.784498    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:11.064916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.064884    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"]
Oct 30 06:08:11.065486 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:08:11.065459    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad85965_824b_4e18_8cb7_3df26685c0ff.slice/crio-ec56036303a10cc7209121387c074ce724acdac9fe49baea456877c601cc29f0 WatchSource:0}: Error finding container ec56036303a10cc7209121387c074ce724acdac9fe49baea456877c601cc29f0: Status 404 returned error can't find the container with id ec56036303a10cc7209121387c074ce724acdac9fe49baea456877c601cc29f0
Oct 30 06:08:11.285960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.285922    5250 generic.go:356] "Generic (PLEG): container finished" podID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerID="aa20b2223e0ff59efc9bc28f9fa9e7b0ff63ead7c54738ef7215e3287a4758da" exitCode=0
Oct 30 06:08:11.288752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.288230    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" event={"ID":"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2","Type":"ContainerDied","Data":"aa20b2223e0ff59efc9bc28f9fa9e7b0ff63ead7c54738ef7215e3287a4758da"}
Oct 30 06:08:11.288752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.288363    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd" event={"ID":"8ad85965-824b-4e18-8cb7-3df26685c0ff","Type":"ContainerStarted","Data":"ec56036303a10cc7209121387c074ce724acdac9fe49baea456877c601cc29f0"}
Oct 30 06:08:11.327820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.327791    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 06:08:11.465506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465469    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-data\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.465506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465511    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-certs\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.465812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465557    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-kube-api-access-tgkzn\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-kube-api-access-tgkzn\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.465812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465617    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-credential-socket\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.465812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465706    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-podinfo\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.465812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465750    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-data" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:11.465812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465780    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-socket\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.466091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465823    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-envoy\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.466091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465833    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-certs" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:11.466091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465857    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istiod-ca-cert\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.466091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465892    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-token\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-token\") pod \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\" (UID: \"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2\") "
Oct 30 06:08:11.466091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.465975    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-credential-socket" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:11.466091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.466062    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-socket" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:11.466452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.466270    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istiod-ca-cert" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:11.466620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.466589    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:11.466620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.466613    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:11.466620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.466629    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:11.466620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.466642    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:11.466620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.466655    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:11.555917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.555869    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-envoy" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:11.556222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.556191    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-podinfo" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:08:11.556311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.556226    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-token" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:11.556311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.556267    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-kube-api-access-tgkzn" pod "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" (UID: "4d5854ee-8d3d-4429-9eb4-f1b3d11861f2"). InnerVolumeSpecName "kube-api-access-tgkzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:11.568725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.568679    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:11.568725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.568701    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:11.568725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.568710    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:11.568725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:11.568719    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgkzn\" (UniqueName: \"kubernetes.io/projected/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2-kube-api-access-tgkzn\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:12.298703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:12.298663    5250 generic.go:356] "Generic (PLEG): container finished" podID="8ad85965-824b-4e18-8cb7-3df26685c0ff" containerID="b69ee1174d0951743560f835eba881b6ab6549711dc3adeb8813c11e98b824fa" exitCode=0
Oct 30 06:08:12.299220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:12.298751    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd" event={"ID":"8ad85965-824b-4e18-8cb7-3df26685c0ff","Type":"ContainerDied","Data":"b69ee1174d0951743560f835eba881b6ab6549711dc3adeb8813c11e98b824fa"}
Oct 30 06:08:12.300946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:12.300920    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"
Oct 30 06:08:12.301077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:12.300941    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf" event={"ID":"4d5854ee-8d3d-4429-9eb4-f1b3d11861f2","Type":"ContainerDied","Data":"bb328b6b1ba8381509e3339f89153cb86c56c4473fc5c88fa2477bba156452d8"}
Oct 30 06:08:12.301077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:12.301011    5250 scope.go:122] "RemoveContainer" containerID="aa20b2223e0ff59efc9bc28f9fa9e7b0ff63ead7c54738ef7215e3287a4758da"
Oct 30 06:08:12.372419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:12.372386    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"]
Oct 30 06:08:12.382586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:12.382559    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-cctsf"]
Oct 30 06:08:12.746250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:12.746224    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:13.049605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.049566    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"]
Oct 30 06:08:13.071748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.071718    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"]
Oct 30 06:08:13.073380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.073336    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 06:08:13.105763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.105737    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 06:08:13.189522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.189269    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd834ca2-0b67-496c-9cb0-bab44ac5e884-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd834ca2-0b67-496c-9cb0-bab44ac5e884-plugins-config-volume\") pod \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\" (UID: \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\") "
Oct 30 06:08:13.189522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.189370    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd834ca2-0b67-496c-9cb0-bab44ac5e884-kube-api-access-skjpm\" (UniqueName: \"kubernetes.io/projected/dd834ca2-0b67-496c-9cb0-bab44ac5e884-kube-api-access-skjpm\") pod \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\" (UID: \"dd834ca2-0b67-496c-9cb0-bab44ac5e884\") "
Oct 30 06:08:13.189522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.189416    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-plugins-config-volume\") pod \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\" (UID: \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\") "
Oct 30 06:08:13.189522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.189516    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-kube-api-access-vssg4\" (UniqueName: \"kubernetes.io/projected/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-kube-api-access-vssg4\") pod \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\" (UID: \"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2\") "
Oct 30 06:08:13.189903 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.189819    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd834ca2-0b67-496c-9cb0-bab44ac5e884-plugins-config-volume" pod "dd834ca2-0b67-496c-9cb0-bab44ac5e884" (UID: "dd834ca2-0b67-496c-9cb0-bab44ac5e884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:13.189976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.189911    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-plugins-config-volume" pod "7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" (UID: "7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:13.191712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.191680    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd834ca2-0b67-496c-9cb0-bab44ac5e884-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:13.191712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.191714    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:13.193425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.193384    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd834ca2-0b67-496c-9cb0-bab44ac5e884-kube-api-access-skjpm" pod "dd834ca2-0b67-496c-9cb0-bab44ac5e884" (UID: "dd834ca2-0b67-496c-9cb0-bab44ac5e884"). InnerVolumeSpecName "kube-api-access-skjpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:13.193569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.193427    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-kube-api-access-vssg4" pod "7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" (UID: "7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2"). InnerVolumeSpecName "kube-api-access-vssg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:13.213264 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.213235    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:13.293985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.293882    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skjpm\" (UniqueName: \"kubernetes.io/projected/dd834ca2-0b67-496c-9cb0-bab44ac5e884-kube-api-access-skjpm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:13.293985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.293941    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vssg4\" (UniqueName: \"kubernetes.io/projected/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2-kube-api-access-vssg4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:13.308263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.308231    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" path="/var/lib/kubelet/pods/4d5854ee-8d3d-4429-9eb4-f1b3d11861f2/volumes"
Oct 30 06:08:13.314009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.313977    5250 generic.go:356] "Generic (PLEG): container finished" podID="7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" containerID="e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff" exitCode=0
Oct 30 06:08:13.314209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.314035    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"
Oct 30 06:08:13.314209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.314052    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5" event={"ID":"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2","Type":"ContainerDied","Data":"e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff"}
Oct 30 06:08:13.314209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.314104    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5" event={"ID":"7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2","Type":"ContainerDied","Data":"466c210f15d0cce685920aa2939ccfc57c31f951f216024774db03f6a0a6be03"}
Oct 30 06:08:13.314209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.314125    5250 scope.go:122] "RemoveContainer" containerID="e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff"
Oct 30 06:08:13.317352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.317004    5250 generic.go:356] "Generic (PLEG): container finished" podID="dd834ca2-0b67-496c-9cb0-bab44ac5e884" containerID="0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4" exitCode=0
Oct 30 06:08:13.317352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.317042    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"
Oct 30 06:08:13.317352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.317056    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc" event={"ID":"dd834ca2-0b67-496c-9cb0-bab44ac5e884","Type":"ContainerDied","Data":"0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4"}
Oct 30 06:08:13.317352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.317127    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc" event={"ID":"dd834ca2-0b67-496c-9cb0-bab44ac5e884","Type":"ContainerDied","Data":"16be956f8857258e605e58d6dc4d97b0f893149d0358fbb82d95c6231bc789e1"}
Oct 30 06:08:13.330241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.330172    5250 scope.go:122] "RemoveContainer" containerID="e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff"
Oct 30 06:08:13.330630 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:13.330609    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff\": container with ID starting with e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff not found: ID does not exist" containerID="e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff"
Oct 30 06:08:13.330716 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.330638    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff"} err="failed to get container status \"e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff\": rpc error: code = NotFound desc = could not find container \"e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff\": container with ID starting with e59bbc0b1d5df5b4f8ec186d8c3d756670688f76679ca45acd60bb9402dcaaff not found: ID does not exist"
Oct 30 06:08:13.330716 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.330657    5250 scope.go:122] "RemoveContainer" containerID="0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4"
Oct 30 06:08:13.341557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.341538    5250 scope.go:122] "RemoveContainer" containerID="0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4"
Oct 30 06:08:13.341889 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:13.341864    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4\": container with ID starting with 0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4 not found: ID does not exist" containerID="0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4"
Oct 30 06:08:13.341965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.341897    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4"} err="failed to get container status \"0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4\": rpc error: code = NotFound desc = could not find container \"0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4\": container with ID starting with 0af51a6fd757629a24e74d4f103ffc26ff181a6af748f94ff1f00f86696e13c4 not found: ID does not exist"
Oct 30 06:08:13.364394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.364356    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"]
Oct 30 06:08:13.370470 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.370301    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:13.378234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.376406    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dbaf6-685586d4dd-wg8nc"]
Oct 30 06:08:13.406838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.406810    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"]
Oct 30 06:08:13.418151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.418112    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dbaf6-6d5948bc8c-czmv5"]
Oct 30 06:08:13.481784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.481762    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:13.597948 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.597862    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad85965-824b-4e18-8cb7-3df26685c0ff-kube-api-access-q4ncj\" (UniqueName: \"kubernetes.io/projected/8ad85965-824b-4e18-8cb7-3df26685c0ff-kube-api-access-q4ncj\") pod \"8ad85965-824b-4e18-8cb7-3df26685c0ff\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") "
Oct 30 06:08:13.598126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.597960    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-xks-ca\") pod \"8ad85965-824b-4e18-8cb7-3df26685c0ff\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") "
Oct 30 06:08:13.598126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.598012    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-openshift-ca\") pod \"8ad85965-824b-4e18-8cb7-3df26685c0ff\" (UID: \"8ad85965-824b-4e18-8cb7-3df26685c0ff\") "
Oct 30 06:08:13.598450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.598420    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-openshift-ca" pod "8ad85965-824b-4e18-8cb7-3df26685c0ff" (UID: "8ad85965-824b-4e18-8cb7-3df26685c0ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:13.598577 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.598528    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:13.600404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.600380    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad85965-824b-4e18-8cb7-3df26685c0ff-kube-api-access-q4ncj" pod "8ad85965-824b-4e18-8cb7-3df26685c0ff" (UID: "8ad85965-824b-4e18-8cb7-3df26685c0ff"). InnerVolumeSpecName "kube-api-access-q4ncj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:13.600507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.600445    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-xks-ca" pod "8ad85965-824b-4e18-8cb7-3df26685c0ff" (UID: "8ad85965-824b-4e18-8cb7-3df26685c0ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:13.702201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.701586    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4ncj\" (UniqueName: \"kubernetes.io/projected/8ad85965-824b-4e18-8cb7-3df26685c0ff-kube-api-access-q4ncj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:13.702201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:13.701629    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad85965-824b-4e18-8cb7-3df26685c0ff-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:14.327040 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:14.327006    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"
Oct 30 06:08:14.327552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:14.327015    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd" event={"ID":"8ad85965-824b-4e18-8cb7-3df26685c0ff","Type":"ContainerDied","Data":"ec56036303a10cc7209121387c074ce724acdac9fe49baea456877c601cc29f0"}
Oct 30 06:08:14.327552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:14.327138    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec56036303a10cc7209121387c074ce724acdac9fe49baea456877c601cc29f0"
Oct 30 06:08:14.597768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:14.597670    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:15.293719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.293684    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" path="/var/lib/kubelet/pods/7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2/volumes"
Oct 30 06:08:15.294145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.294130    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd834ca2-0b67-496c-9cb0-bab44ac5e884" path="/var/lib/kubelet/pods/dd834ca2-0b67-496c-9cb0-bab44ac5e884/volumes"
Oct 30 06:08:15.589122 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.589034    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"]
Oct 30 06:08:15.590013 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.589990    5250 cpu_manager.go:414] "containerMap: removing container" podUID="7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" containerName="payload-pre-processing"
Oct 30 06:08:15.590140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590014    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" containerName="payload-pre-processing"
Oct 30 06:08:15.590140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590031    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy"
Oct 30 06:08:15.590140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590039    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy"
Oct 30 06:08:15.590140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590074    5250 cpu_manager.go:414] "containerMap: removing container" podUID="dd834ca2-0b67-496c-9cb0-bab44ac5e884" containerName="payload-processing"
Oct 30 06:08:15.590140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590082    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd834ca2-0b67-496c-9cb0-bab44ac5e884" containerName="payload-processing"
Oct 30 06:08:15.590140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590103    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8ad85965-824b-4e18-8cb7-3df26685c0ff" containerName="revoke-keys"
Oct 30 06:08:15.590140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590110    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad85965-824b-4e18-8cb7-3df26685c0ff" containerName="revoke-keys"
Oct 30 06:08:15.590501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590318    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3d61d5-85f2-4ef2-81ff-e2a0b320a2c2" containerName="payload-pre-processing"
Oct 30 06:08:15.590501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590338    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5854ee-8d3d-4429-9eb4-f1b3d11861f2" containerName="istio-proxy"
Oct 30 06:08:15.590501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590352    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd834ca2-0b67-496c-9cb0-bab44ac5e884" containerName="payload-processing"
Oct 30 06:08:15.590501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.590363    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad85965-824b-4e18-8cb7-3df26685c0ff" containerName="revoke-keys"
Oct 30 06:08:15.595612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.595593    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.612316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.612292    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"]
Oct 30 06:08:15.616567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.616541    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1b4b7a\"/\"default-dockercfg-wlbgp\""
Oct 30 06:08:15.616727 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.616707    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-1b4b7a\"/\"rate-test-model-1b4b7a-kserve-self-signed-certs\""
Oct 30 06:08:15.620887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.620868    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-1b4b7a\"/\"kube-root-ca.crt\""
Oct 30 06:08:15.620984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.620886    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-1b4b7a\"/\"openshift-service-ca.crt\""
Oct 30 06:08:15.667864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.667831    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:15.737699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.737666    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-tmp-dir\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.737885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.737705    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-dshm\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.737885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.737747    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-home\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.737976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.737900    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-model-cache\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.738040 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.738010    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqs25\" (UniqueName: \"kubernetes.io/projected/b9e94010-987c-4a49-bb42-3a1dd7a495da-kube-api-access-fqs25\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.738172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.738155    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e94010-987c-4a49-bb42-3a1dd7a495da-tls-certs\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.845265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.844243    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-tmp-dir\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.845265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.844302    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-dshm\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.845265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.844365    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-home\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.845265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.844398    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-model-cache\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.845265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.844456    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqs25\" (UniqueName: \"kubernetes.io/projected/b9e94010-987c-4a49-bb42-3a1dd7a495da-kube-api-access-fqs25\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.845265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.844588    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e94010-987c-4a49-bb42-3a1dd7a495da-tls-certs\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.845719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.845596    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-home\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.856102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.845838    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-tmp-dir\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.856102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.846374    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-model-cache\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.859050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.859021    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-dshm\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.859917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.859894    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e94010-987c-4a49-bb42-3a1dd7a495da-tls-certs\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.871356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.868993    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqs25\" (UniqueName: \"kubernetes.io/projected/b9e94010-987c-4a49-bb42-3a1dd7a495da-kube-api-access-fqs25\") pod \"rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") " pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:15.918594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:15.918556    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:16.269416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:16.269391    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"]
Oct 30 06:08:16.270741 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:08:16.270714    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9e94010_987c_4a49_bb42_3a1dd7a495da.slice/crio-e896232e288d68900214595306b41a1fefee7870d6b1568cbbb294467e7db756 WatchSource:0}: Error finding container e896232e288d68900214595306b41a1fefee7870d6b1568cbbb294467e7db756: Status 404 returned error can't find the container with id e896232e288d68900214595306b41a1fefee7870d6b1568cbbb294467e7db756
Oct 30 06:08:16.349119 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:16.349090    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4" event={"ID":"b9e94010-987c-4a49-bb42-3a1dd7a495da","Type":"ContainerStarted","Data":"ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8"}
Oct 30 06:08:16.349298 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:16.349131    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4" event={"ID":"b9e94010-987c-4a49-bb42-3a1dd7a495da","Type":"ContainerStarted","Data":"e896232e288d68900214595306b41a1fefee7870d6b1568cbbb294467e7db756"}
Oct 30 06:08:16.395051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:16.395005    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:16.399074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:16.398975    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4" podStartSLOduration=1.398955853 podStartE2EDuration="1.398955853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:16.393134852 +0000 UTC m=+2211.224172072" watchObservedRunningTime="2026-10-30 06:08:16.398955853 +0000 UTC m=+2211.229993070"
Oct 30 06:08:16.919748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:16.919710    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:18.050914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:18.050879    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj" podUID="8fad536f-7c39-4463-a60d-e34aae5fc8de" containerName="payload-processing" containerID="cri-o://63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce" gracePeriod=25
Oct 30 06:08:18.072790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:18.072748    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd" podUID="52574608-d7bd-49b1-a833-e81372b261c0" containerName="payload-pre-processing" containerID="cri-o://8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413" gracePeriod=25
Oct 30 06:08:18.269363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:18.269332    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:19.980320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:19.980273    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:20.237547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.237455    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj" podUID="8fad536f-7c39-4463-a60d-e34aae5fc8de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:20.816904 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.816871    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"]
Oct 30 06:08:20.824995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.824965    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:20.845508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.845473    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"]
Oct 30 06:08:20.848431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.848406    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-452a59\"/\"kube-root-ca.crt\""
Oct 30 06:08:20.848662 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.848637    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-452a59\"/\"openshift-service-ca.crt\""
Oct 30 06:08:20.850644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.850626    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-452a59\"/\"default-dockercfg-d2q44\""
Oct 30 06:08:20.850766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.850669    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-452a59\"/\"sub-test-model-452a59-kserve-self-signed-certs\""
Oct 30 06:08:20.910502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.910465    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:20.920370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.920335    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5zn5\" (UniqueName: \"kubernetes.io/projected/762fb9b3-a800-478d-bb49-9180d8ece9be-kube-api-access-s5zn5\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:20.920571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.920395    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-tmp-dir\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:20.920571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.920489    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-home\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:20.920571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.920525    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-model-cache\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:20.920571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.920556    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-dshm\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:20.920775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:20.920736    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762fb9b3-a800-478d-bb49-9180d8ece9be-tls-certs\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.022348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.022294    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5zn5\" (UniqueName: \"kubernetes.io/projected/762fb9b3-a800-478d-bb49-9180d8ece9be-kube-api-access-s5zn5\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.022845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.022399    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-tmp-dir\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.022845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.022483    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-home\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.022845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.022528    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-model-cache\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.022845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.022590    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-dshm\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.022845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.022774    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762fb9b3-a800-478d-bb49-9180d8ece9be-tls-certs\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.023142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.022862    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-tmp-dir\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.023142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.022895    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-home\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.023566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.023543    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-model-cache\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.025455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.025431    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-dshm\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.025629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.025609    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762fb9b3-a800-478d-bb49-9180d8ece9be-tls-certs\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.044670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.044639    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5zn5\" (UniqueName: \"kubernetes.io/projected/762fb9b3-a800-478d-bb49-9180d8ece9be-kube-api-access-s5zn5\") pod \"sub-test-model-452a59-kserve-676b9d4b8f-9jng8\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") " pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.158245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.155815    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:21.263919 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.263861    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd" podUID="52574608-d7bd-49b1-a833-e81372b261c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:21.529709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.529678    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"]
Oct 30 06:08:21.530335 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:08:21.530308    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod762fb9b3_a800_478d_bb49_9180d8ece9be.slice/crio-13ea19bc0671210ec882dcf7b9e1387439beec6e659cc12741d42c45d2b791e0 WatchSource:0}: Error finding container 13ea19bc0671210ec882dcf7b9e1387439beec6e659cc12741d42c45d2b791e0: Status 404 returned error can't find the container with id 13ea19bc0671210ec882dcf7b9e1387439beec6e659cc12741d42c45d2b791e0
Oct 30 06:08:21.736221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:21.736150    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:22.400945 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:22.400911    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8" event={"ID":"762fb9b3-a800-478d-bb49-9180d8ece9be","Type":"ContainerStarted","Data":"4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00"}
Oct 30 06:08:22.401373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:22.400956    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8" event={"ID":"762fb9b3-a800-478d-bb49-9180d8ece9be","Type":"ContainerStarted","Data":"13ea19bc0671210ec882dcf7b9e1387439beec6e659cc12741d42c45d2b791e0"}
Oct 30 06:08:22.439753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:22.439705    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8" podStartSLOduration=2.439691393 podStartE2EDuration="2.439691393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:22.437599315 +0000 UTC m=+2217.268636535" watchObservedRunningTime="2026-10-30 06:08:22.439691393 +0000 UTC m=+2217.270728613"
Oct 30 06:08:23.156472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:23.156443    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:23.493850 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:23.493769    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:25.189397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:25.189355    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:25.919507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:25.919478    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:25.932503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:25.932471    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:26.451000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:26.450973    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:08:26.886867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:26.886830    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:28.776599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:28.776565    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:29.092982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.092870    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:29.567530 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.567496    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"]
Oct 30 06:08:29.573849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.573827    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.594482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.594444    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"]
Oct 30 06:08:29.598834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.598804    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:08:29.598973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.598956    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:08:29.656792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.656750    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.657405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.657378    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthjx\" (UniqueName: \"kubernetes.io/projected/0ff983fd-753b-464a-8655-f5c9b84665cc-kube-api-access-gthjx\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.657593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.657579    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.759016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.758983    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthjx\" (UniqueName: \"kubernetes.io/projected/0ff983fd-753b-464a-8655-f5c9b84665cc-kube-api-access-gthjx\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.759226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.759020    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.759226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.759105    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.759844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.759821    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.761772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.761754    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.780820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.780801    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthjx\" (UniqueName: \"kubernetes.io/projected/0ff983fd-753b-464a-8655-f5c9b84665cc-kube-api-access-gthjx\") pod \"maas-api-revoke-sub-df18371e-8vptp\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:29.899508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:29.899483    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:30.211790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:30.211762    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"]
Oct 30 06:08:30.212375 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:08:30.212350    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff983fd_753b_464a_8655_f5c9b84665cc.slice/crio-56b2dc76e3df8ccb61a50d7627d38da61605dc60bc91ded38042a9d656f7f879 WatchSource:0}: Error finding container 56b2dc76e3df8ccb61a50d7627d38da61605dc60bc91ded38042a9d656f7f879: Status 404 returned error can't find the container with id 56b2dc76e3df8ccb61a50d7627d38da61605dc60bc91ded38042a9d656f7f879
Oct 30 06:08:30.237933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:30.237898    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj" podUID="8fad536f-7c39-4463-a60d-e34aae5fc8de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:30.471231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:30.471170    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp" event={"ID":"0ff983fd-753b-464a-8655-f5c9b84665cc","Type":"ContainerStarted","Data":"56b2dc76e3df8ccb61a50d7627d38da61605dc60bc91ded38042a9d656f7f879"}
Oct 30 06:08:30.475894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:30.475868    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:31.156965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:31.156933    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:31.169674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:31.169645    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:31.261781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:31.261746    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd" podUID="52574608-d7bd-49b1-a833-e81372b261c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:31.481441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:31.481358    5250 generic.go:356] "Generic (PLEG): container finished" podID="0ff983fd-753b-464a-8655-f5c9b84665cc" containerID="d05170c3baf1a3221963be4f2f1ee7ee2b1d363e1ec369a1368a4a1fe87cfbcf" exitCode=0
Oct 30 06:08:31.481620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:31.481446    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp" event={"ID":"0ff983fd-753b-464a-8655-f5c9b84665cc","Type":"ContainerDied","Data":"d05170c3baf1a3221963be4f2f1ee7ee2b1d363e1ec369a1368a4a1fe87cfbcf"}
Oct 30 06:08:31.494321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:31.494286    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:08:32.119942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.119905    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:32.374416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.374332    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"]
Oct 30 06:08:32.404973 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:32.404922    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:08:32.426841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.426809    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:32.793706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.793678    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:32.967591 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.967504    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff983fd-753b-464a-8655-f5c9b84665cc-kube-api-access-gthjx\" (UniqueName: \"kubernetes.io/projected/0ff983fd-753b-464a-8655-f5c9b84665cc-kube-api-access-gthjx\") pod \"0ff983fd-753b-464a-8655-f5c9b84665cc\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") "
Oct 30 06:08:32.967749 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.967594    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-xks-ca\") pod \"0ff983fd-753b-464a-8655-f5c9b84665cc\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") "
Oct 30 06:08:32.967905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.967888    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-openshift-ca\") pod \"0ff983fd-753b-464a-8655-f5c9b84665cc\" (UID: \"0ff983fd-753b-464a-8655-f5c9b84665cc\") "
Oct 30 06:08:32.968327 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.968302    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-openshift-ca" pod "0ff983fd-753b-464a-8655-f5c9b84665cc" (UID: "0ff983fd-753b-464a-8655-f5c9b84665cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:32.968565 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.968550    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:32.969834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.969798    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-xks-ca" pod "0ff983fd-753b-464a-8655-f5c9b84665cc" (UID: "0ff983fd-753b-464a-8655-f5c9b84665cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:32.969940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:32.969867    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff983fd-753b-464a-8655-f5c9b84665cc-kube-api-access-gthjx" pod "0ff983fd-753b-464a-8655-f5c9b84665cc" (UID: "0ff983fd-753b-464a-8655-f5c9b84665cc"). InnerVolumeSpecName "kube-api-access-gthjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:33.070990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:33.070958    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthjx\" (UniqueName: \"kubernetes.io/projected/0ff983fd-753b-464a-8655-f5c9b84665cc-kube-api-access-gthjx\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:33.071353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:33.071311    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff983fd-753b-464a-8655-f5c9b84665cc-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:33.553417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:33.553321    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp" event={"ID":"0ff983fd-753b-464a-8655-f5c9b84665cc","Type":"ContainerDied","Data":"56b2dc76e3df8ccb61a50d7627d38da61605dc60bc91ded38042a9d656f7f879"}
Oct 30 06:08:33.553417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:33.553341    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"
Oct 30 06:08:33.553417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:33.553369    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56b2dc76e3df8ccb61a50d7627d38da61605dc60bc91ded38042a9d656f7f879"
Oct 30 06:08:33.809787 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:33.809698    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:34.740665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:34.740628    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:34.876214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:34.876163    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"]
Oct 30 06:08:34.881971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:34.881935    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:34.885868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:34.885841    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dbaf6-dwl2b"]
Oct 30 06:08:35.028879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:35.028791    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:35.282891 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:35.282813    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ba6160-f995-4baa-a187-6ed272669a84" path="/var/lib/kubelet/pods/e5ba6160-f995-4baa-a187-6ed272669a84/volumes"
Oct 30 06:08:35.421885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:35.421857    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:35.563556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:35.563459    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:36.895337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:36.895305    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:38.385944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.385917    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:08:38.411494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.411473    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:08:38.473878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.473840    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fad536f-7c39-4463-a60d-e34aae5fc8de-kube-api-access-qzjd6\" (UniqueName: \"kubernetes.io/projected/8fad536f-7c39-4463-a60d-e34aae5fc8de-kube-api-access-qzjd6\") pod \"8fad536f-7c39-4463-a60d-e34aae5fc8de\" (UID: \"8fad536f-7c39-4463-a60d-e34aae5fc8de\") "
Oct 30 06:08:38.474053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.473934    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52574608-d7bd-49b1-a833-e81372b261c0-kube-api-access-ljk6t\" (UniqueName: \"kubernetes.io/projected/52574608-d7bd-49b1-a833-e81372b261c0-kube-api-access-ljk6t\") pod \"52574608-d7bd-49b1-a833-e81372b261c0\" (UID: \"52574608-d7bd-49b1-a833-e81372b261c0\") "
Oct 30 06:08:38.474053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.473976    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52574608-d7bd-49b1-a833-e81372b261c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52574608-d7bd-49b1-a833-e81372b261c0-plugins-config-volume\") pod \"52574608-d7bd-49b1-a833-e81372b261c0\" (UID: \"52574608-d7bd-49b1-a833-e81372b261c0\") "
Oct 30 06:08:38.474218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.474070    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fad536f-7c39-4463-a60d-e34aae5fc8de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fad536f-7c39-4463-a60d-e34aae5fc8de-plugins-config-volume\") pod \"8fad536f-7c39-4463-a60d-e34aae5fc8de\" (UID: \"8fad536f-7c39-4463-a60d-e34aae5fc8de\") "
Oct 30 06:08:38.474984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.474947    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52574608-d7bd-49b1-a833-e81372b261c0-plugins-config-volume" pod "52574608-d7bd-49b1-a833-e81372b261c0" (UID: "52574608-d7bd-49b1-a833-e81372b261c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:38.475106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.475020    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fad536f-7c39-4463-a60d-e34aae5fc8de-plugins-config-volume" pod "8fad536f-7c39-4463-a60d-e34aae5fc8de" (UID: "8fad536f-7c39-4463-a60d-e34aae5fc8de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:38.477168 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.477139    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52574608-d7bd-49b1-a833-e81372b261c0-kube-api-access-ljk6t" pod "52574608-d7bd-49b1-a833-e81372b261c0" (UID: "52574608-d7bd-49b1-a833-e81372b261c0"). InnerVolumeSpecName "kube-api-access-ljk6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:38.478894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.478866    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fad536f-7c39-4463-a60d-e34aae5fc8de-kube-api-access-qzjd6" pod "8fad536f-7c39-4463-a60d-e34aae5fc8de" (UID: "8fad536f-7c39-4463-a60d-e34aae5fc8de"). InnerVolumeSpecName "kube-api-access-qzjd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:38.576313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.576231    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzjd6\" (UniqueName: \"kubernetes.io/projected/8fad536f-7c39-4463-a60d-e34aae5fc8de-kube-api-access-qzjd6\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:38.576313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.576257    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljk6t\" (UniqueName: \"kubernetes.io/projected/52574608-d7bd-49b1-a833-e81372b261c0-kube-api-access-ljk6t\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:38.576313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.576268    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52574608-d7bd-49b1-a833-e81372b261c0-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:38.576313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.576285    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fad536f-7c39-4463-a60d-e34aae5fc8de-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:38.597878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.597850    5250 generic.go:356] "Generic (PLEG): container finished" podID="52574608-d7bd-49b1-a833-e81372b261c0" containerID="8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413" exitCode=0
Oct 30 06:08:38.598072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.597926    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd" event={"ID":"52574608-d7bd-49b1-a833-e81372b261c0","Type":"ContainerDied","Data":"8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413"}
Oct 30 06:08:38.598072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.597969    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"
Oct 30 06:08:38.598072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.597995    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd" event={"ID":"52574608-d7bd-49b1-a833-e81372b261c0","Type":"ContainerDied","Data":"5c8ec43ae385f73e29f9c09cd78800426352981cf0c5ced66ff15a2a6ced9158"}
Oct 30 06:08:38.598072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.598027    5250 scope.go:122] "RemoveContainer" containerID="8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413"
Oct 30 06:08:38.599652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.599633    5250 generic.go:356] "Generic (PLEG): container finished" podID="8fad536f-7c39-4463-a60d-e34aae5fc8de" containerID="63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce" exitCode=0
Oct 30 06:08:38.599792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.599667    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj" event={"ID":"8fad536f-7c39-4463-a60d-e34aae5fc8de","Type":"ContainerDied","Data":"63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce"}
Oct 30 06:08:38.599792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.599688    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"
Oct 30 06:08:38.599792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.599701    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj" event={"ID":"8fad536f-7c39-4463-a60d-e34aae5fc8de","Type":"ContainerDied","Data":"e773bae755fa32875181aa2afe8d7d2d6a632235cd8fef786d3559a500b0436c"}
Oct 30 06:08:38.610705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.610681    5250 scope.go:122] "RemoveContainer" containerID="8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413"
Oct 30 06:08:38.611021 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:38.610998    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413\": container with ID starting with 8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413 not found: ID does not exist" containerID="8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413"
Oct 30 06:08:38.611123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.611034    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413"} err="failed to get container status \"8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413\": rpc error: code = NotFound desc = could not find container \"8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413\": container with ID starting with 8b2f66013be8ac8055996a7a1943182387500c4bbcce0c142dd26efcff408413 not found: ID does not exist"
Oct 30 06:08:38.611123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.611057    5250 scope.go:122] "RemoveContainer" containerID="63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce"
Oct 30 06:08:38.622105 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.622082    5250 scope.go:122] "RemoveContainer" containerID="63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce"
Oct 30 06:08:38.622399 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:38.622375    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce\": container with ID starting with 63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce not found: ID does not exist" containerID="63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce"
Oct 30 06:08:38.622461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.622411    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce"} err="failed to get container status \"63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce\": rpc error: code = NotFound desc = could not find container \"63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce\": container with ID starting with 63ab7bcda16325ceba5722b2292cc1fbcc99cdbe492caace3ca4caea42d887ce not found: ID does not exist"
Oct 30 06:08:38.646981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.646952    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"]
Oct 30 06:08:38.653098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.653075    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8de592-7dcc7c7555-mfwfd"]
Oct 30 06:08:38.684407 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.684376    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"]
Oct 30 06:08:38.688689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.688667    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8de592-58c4dc884d-k5rsj"]
Oct 30 06:08:38.713699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:38.713658    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:39.282087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:39.282050    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52574608-d7bd-49b1-a833-e81372b261c0" path="/var/lib/kubelet/pods/52574608-d7bd-49b1-a833-e81372b261c0/volumes"
Oct 30 06:08:39.282663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:39.282641    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fad536f-7c39-4463-a60d-e34aae5fc8de" path="/var/lib/kubelet/pods/8fad536f-7c39-4463-a60d-e34aae5fc8de/volumes"
Oct 30 06:08:40.460824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:40.459382    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:41.828444 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:41.828412    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:41.987929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:41.987900    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:42.214409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.214374    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"]
Oct 30 06:08:42.214665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.214641    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy" containerID="cri-o://e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682" gracePeriod=30
Oct 30 06:08:42.265225 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.265173    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:42.419369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.415693    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:42.822464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.822424    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"]
Oct 30 06:08:42.823149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823133    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0ff983fd-753b-464a-8655-f5c9b84665cc" containerName="revoke-keys"
Oct 30 06:08:42.823320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823151    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff983fd-753b-464a-8655-f5c9b84665cc" containerName="revoke-keys"
Oct 30 06:08:42.823320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823164    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8fad536f-7c39-4463-a60d-e34aae5fc8de" containerName="payload-processing"
Oct 30 06:08:42.823320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823169    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fad536f-7c39-4463-a60d-e34aae5fc8de" containerName="payload-processing"
Oct 30 06:08:42.823320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823222    5250 cpu_manager.go:414] "containerMap: removing container" podUID="52574608-d7bd-49b1-a833-e81372b261c0" containerName="payload-pre-processing"
Oct 30 06:08:42.823320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823227    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52574608-d7bd-49b1-a833-e81372b261c0" containerName="payload-pre-processing"
Oct 30 06:08:42.823486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823332    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="52574608-d7bd-49b1-a833-e81372b261c0" containerName="payload-pre-processing"
Oct 30 06:08:42.823486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823341    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff983fd-753b-464a-8655-f5c9b84665cc" containerName="revoke-keys"
Oct 30 06:08:42.823486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.823349    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fad536f-7c39-4463-a60d-e34aae5fc8de" containerName="payload-processing"
Oct 30 06:08:42.828426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.828410    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:42.845483 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.845459    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"]
Oct 30 06:08:42.939851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.939819    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:08:42.940020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.939879    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:08:42.964440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.964411    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:42.964638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.964542    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:42.964638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:42.964622    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lwxs\" (UniqueName: \"kubernetes.io/projected/32b57a2f-7a55-4765-8dcb-52b393877d13-kube-api-access-8lwxs\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:43.066015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.065984    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:43.066198 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.066141    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:43.066613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.066582    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lwxs\" (UniqueName: \"kubernetes.io/projected/32b57a2f-7a55-4765-8dcb-52b393877d13-kube-api-access-8lwxs\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:43.066869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.066846    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:43.069035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.069009    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:43.088093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.088037    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lwxs\" (UniqueName: \"kubernetes.io/projected/32b57a2f-7a55-4765-8dcb-52b393877d13-kube-api-access-8lwxs\") pod \"maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:43.142627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.142597    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:43.345093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.345016    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"]
Oct 30 06:08:43.370720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.370680    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"]
Oct 30 06:08:43.459450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.458709    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"]
Oct 30 06:08:43.498289 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.498251    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:43.646344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.646313    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2" event={"ID":"32b57a2f-7a55-4765-8dcb-52b393877d13","Type":"ContainerStarted","Data":"d709363fd550b2feb6b930d128a6c3b312db4ba46d2ab8143801354b3d752fa0"}
Oct 30 06:08:43.646591 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:43.646380    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:44.660101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:44.660061    5250 generic.go:356] "Generic (PLEG): container finished" podID="32b57a2f-7a55-4765-8dcb-52b393877d13" containerID="2543d8c52c73d3e6d3927f2be75b44fc6068e8d59d0a13d2685adf18ecf8f668" exitCode=0
Oct 30 06:08:44.660546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:44.660150    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2" event={"ID":"32b57a2f-7a55-4765-8dcb-52b393877d13","Type":"ContainerDied","Data":"2543d8c52c73d3e6d3927f2be75b44fc6068e8d59d0a13d2685adf18ecf8f668"}
Oct 30 06:08:44.685613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:44.685578    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:44.870370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:44.861950    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:44.981366 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:44.981331    5250 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:08:44.981568 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:44.981397    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:08:45.832703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:45.832676    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:45.921373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:45.921287    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-openshift-ca\") pod \"32b57a2f-7a55-4765-8dcb-52b393877d13\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") "
Oct 30 06:08:45.921610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:45.921589    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b57a2f-7a55-4765-8dcb-52b393877d13-kube-api-access-8lwxs\" (UniqueName: \"kubernetes.io/projected/32b57a2f-7a55-4765-8dcb-52b393877d13-kube-api-access-8lwxs\") pod \"32b57a2f-7a55-4765-8dcb-52b393877d13\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") "
Oct 30 06:08:45.921701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:45.921638    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-xks-ca\") pod \"32b57a2f-7a55-4765-8dcb-52b393877d13\" (UID: \"32b57a2f-7a55-4765-8dcb-52b393877d13\") "
Oct 30 06:08:45.921935 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:45.921908    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-openshift-ca" pod "32b57a2f-7a55-4765-8dcb-52b393877d13" (UID: "32b57a2f-7a55-4765-8dcb-52b393877d13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:45.922315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:45.922283    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:45.924878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:45.924847    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-xks-ca" pod "32b57a2f-7a55-4765-8dcb-52b393877d13" (UID: "32b57a2f-7a55-4765-8dcb-52b393877d13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:45.938214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:45.936112    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b57a2f-7a55-4765-8dcb-52b393877d13-kube-api-access-8lwxs" pod "32b57a2f-7a55-4765-8dcb-52b393877d13" (UID: "32b57a2f-7a55-4765-8dcb-52b393877d13"). InnerVolumeSpecName "kube-api-access-8lwxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:46.024122 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:46.024090    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lwxs\" (UniqueName: \"kubernetes.io/projected/32b57a2f-7a55-4765-8dcb-52b393877d13-kube-api-access-8lwxs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:46.024122 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:46.024122    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b57a2f-7a55-4765-8dcb-52b393877d13-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:46.677474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:46.677439    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"
Oct 30 06:08:46.677661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:46.677495    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2" event={"ID":"32b57a2f-7a55-4765-8dcb-52b393877d13","Type":"ContainerDied","Data":"d709363fd550b2feb6b930d128a6c3b312db4ba46d2ab8143801354b3d752fa0"}
Oct 30 06:08:46.677661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:46.677561    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d709363fd550b2feb6b930d128a6c3b312db4ba46d2ab8143801354b3d752fa0"
Oct 30 06:08:46.961618 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:46.961516    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:47.402041 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.401923    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv" podUID="fabeb868-afa6-49c3-b089-5905b37f48e4" containerName="main" containerID="cri-o://6d574e54cd53149b70519e8b35c050742004d5c4079a270533bbd443407ee5c1" gracePeriod=45
Oct 30 06:08:47.548893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.548864    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:08:47.653757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.653725    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-data\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.653909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.653797    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-envoy\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.653909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.653838    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-kube-api-access-tz9zj\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-kube-api-access-tz9zj\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.653909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.653869    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fd3513-4813-4402-b48c-8a5a761c45c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fd3513-4813-4402-b48c-8a5a761c45c9-istiod-ca-cert\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.654104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.653951    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-certs\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.654104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654014    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-data" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.654104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654029    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-podinfo\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.654295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654254    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-token\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-token\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.654368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654340    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-socket\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.654368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654354    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fd3513-4813-4402-b48c-8a5a761c45c9-istiod-ca-cert" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:47.654546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654402    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-credential-socket\") pod \"83fd3513-4813-4402-b48c-8a5a761c45c9\" (UID: \"83fd3513-4813-4402-b48c-8a5a761c45c9\") "
Oct 30 06:08:47.654546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654531    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-certs" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.654749 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654726    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-socket" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.654961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.654932    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-credential-socket" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.655249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.655222    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.655345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.655254    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.655345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.655269    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.655345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.655281    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fd3513-4813-4402-b48c-8a5a761c45c9-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.655345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.655292    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.657271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.657244    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-kube-api-access-tz9zj" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "kube-api-access-tz9zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:47.658311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.658289    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-podinfo" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:08:47.658311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.658310    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-envoy" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.658486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.658333    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-token" pod "83fd3513-4813-4402-b48c-8a5a761c45c9" (UID: "83fd3513-4813-4402-b48c-8a5a761c45c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:47.688969 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.688943    5250 generic.go:356] "Generic (PLEG): container finished" podID="fabeb868-afa6-49c3-b089-5905b37f48e4" containerID="6d574e54cd53149b70519e8b35c050742004d5c4079a270533bbd443407ee5c1" exitCode=0
Oct 30 06:08:47.689125 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.689022    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv" event={"ID":"fabeb868-afa6-49c3-b089-5905b37f48e4","Type":"ContainerDied","Data":"6d574e54cd53149b70519e8b35c050742004d5c4079a270533bbd443407ee5c1"}
Oct 30 06:08:47.690856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.690833    5250 generic.go:356] "Generic (PLEG): container finished" podID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerID="e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682" exitCode=0
Oct 30 06:08:47.690983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.690890    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"
Oct 30 06:08:47.690983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.690915    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" event={"ID":"83fd3513-4813-4402-b48c-8a5a761c45c9","Type":"ContainerDied","Data":"e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682"}
Oct 30 06:08:47.690983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.690973    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z" event={"ID":"83fd3513-4813-4402-b48c-8a5a761c45c9","Type":"ContainerDied","Data":"b1f138937594abb4e4ecb4097c9230849b90bf4366434df9b619c3011ac8083d"}
Oct 30 06:08:47.691154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.691006    5250 scope.go:122] "RemoveContainer" containerID="e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682"
Oct 30 06:08:47.697047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.697025    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:08:47.708208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.708192    5250 scope.go:122] "RemoveContainer" containerID="e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682"
Oct 30 06:08:47.708512 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:47.708496    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682\": container with ID starting with e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682 not found: ID does not exist" containerID="e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682"
Oct 30 06:08:47.708571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.708518    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682"} err="failed to get container status \"e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682\": rpc error: code = NotFound desc = could not find container \"e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682\": container with ID starting with e46af1ca655d4713306eaaea88b1fc9e4a6b693e5c3fdddce286ab5f18d24682 not found: ID does not exist"
Oct 30 06:08:47.757472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757393    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-home\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-home\") pod \"fabeb868-afa6-49c3-b089-5905b37f48e4\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") "
Oct 30 06:08:47.757612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757493    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-model-cache\") pod \"fabeb868-afa6-49c3-b089-5905b37f48e4\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") "
Oct 30 06:08:47.757612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757607    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-dshm\") pod \"fabeb868-afa6-49c3-b089-5905b37f48e4\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") "
Oct 30 06:08:47.757748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757647    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabeb868-afa6-49c3-b089-5905b37f48e4-kube-api-access-ts5kb\" (UniqueName: \"kubernetes.io/projected/fabeb868-afa6-49c3-b089-5905b37f48e4-kube-api-access-ts5kb\") pod \"fabeb868-afa6-49c3-b089-5905b37f48e4\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") "
Oct 30 06:08:47.757748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757681    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-tmp-dir\") pod \"fabeb868-afa6-49c3-b089-5905b37f48e4\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") "
Oct 30 06:08:47.757748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757703    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-home" pod "fabeb868-afa6-49c3-b089-5905b37f48e4" (UID: "fabeb868-afa6-49c3-b089-5905b37f48e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.757977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757754    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fabeb868-afa6-49c3-b089-5905b37f48e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/fabeb868-afa6-49c3-b089-5905b37f48e4-tls-certs\") pod \"fabeb868-afa6-49c3-b089-5905b37f48e4\" (UID: \"fabeb868-afa6-49c3-b089-5905b37f48e4\") "
Oct 30 06:08:47.757977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757772    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-model-cache" pod "fabeb868-afa6-49c3-b089-5905b37f48e4" (UID: "fabeb868-afa6-49c3-b089-5905b37f48e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.757977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.757968    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-tmp-dir" pod "fabeb868-afa6-49c3-b089-5905b37f48e4" (UID: "fabeb868-afa6-49c3-b089-5905b37f48e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.758632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.758609    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.758734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.758635    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.758734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.758652    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.758734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.758666    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz9zj\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-kube-api-access-tz9zj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.758734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.758681    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.758734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.758695    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.758734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.758708    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fd3513-4813-4402-b48c-8a5a761c45c9-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.760128 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.760108    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-dshm" pod "fabeb868-afa6-49c3-b089-5905b37f48e4" (UID: "fabeb868-afa6-49c3-b089-5905b37f48e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:08:47.760288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.760266    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabeb868-afa6-49c3-b089-5905b37f48e4-kube-api-access-ts5kb" pod "fabeb868-afa6-49c3-b089-5905b37f48e4" (UID: "fabeb868-afa6-49c3-b089-5905b37f48e4"). InnerVolumeSpecName "kube-api-access-ts5kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:47.760621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.760607    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabeb868-afa6-49c3-b089-5905b37f48e4-tls-certs" pod "fabeb868-afa6-49c3-b089-5905b37f48e4" (UID: "fabeb868-afa6-49c3-b089-5905b37f48e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:08:47.788606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.788576    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"]
Oct 30 06:08:47.793029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.793005    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-h2b6z"]
Oct 30 06:08:47.859868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.859843    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabeb868-afa6-49c3-b089-5905b37f48e4-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.859868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.859869    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5kb\" (UniqueName: \"kubernetes.io/projected/fabeb868-afa6-49c3-b089-5905b37f48e4-kube-api-access-ts5kb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:47.860043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:47.859878    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabeb868-afa6-49c3-b089-5905b37f48e4-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:48.345723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:48.345682    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k" podUID="8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" containerName="payload-processing" containerID="cri-o://4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136" gracePeriod=25
Oct 30 06:08:48.371751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:48.371721    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262" podUID="fb360e09-c857-4ffb-ae6f-c685c24f56d1" containerName="payload-pre-processing" containerID="cri-o://e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf" gracePeriod=25
Oct 30 06:08:48.700200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:48.700154    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"
Oct 30 06:08:48.700200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:48.700163    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv" event={"ID":"fabeb868-afa6-49c3-b089-5905b37f48e4","Type":"ContainerDied","Data":"a0f0c0b677b7fe880e5cef7fff1abdc518c8b63f1f37c9992b186bceae21c4df"}
Oct 30 06:08:48.700472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:48.700233    5250 scope.go:122] "RemoveContainer" containerID="6d574e54cd53149b70519e8b35c050742004d5c4079a270533bbd443407ee5c1"
Oct 30 06:08:48.748123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:48.748096    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"]
Oct 30 06:08:48.758027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:48.758003    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d917a1/test-model-d917a1-kserve-5d9897965-dsdzv"]
Oct 30 06:08:48.884025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:48.883994    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:49.283475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:49.283444    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" path="/var/lib/kubelet/pods/83fd3513-4813-4402-b48c-8a5a761c45c9/volumes"
Oct 30 06:08:49.285820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:49.283903    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabeb868-afa6-49c3-b089-5905b37f48e4" path="/var/lib/kubelet/pods/fabeb868-afa6-49c3-b089-5905b37f48e4/volumes"
Oct 30 06:08:50.493203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:50.492882    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:51.119086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:51.119045    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:51.230643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:51.230610    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"]
Oct 30 06:08:51.241601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:51.241569    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8de592-jqs2z"]
Oct 30 06:08:51.261327 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:51.261288    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:51.282871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:51.282843    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d29b7a-2253-49d8-bd07-2f30c1ae5a54" path="/var/lib/kubelet/pods/e5d29b7a-2253-49d8-bd07-2f30c1ae5a54/volumes"
Oct 30 06:08:51.419218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:51.417220    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:51.806381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:51.806290    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:52.506148 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:52.506116    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:53.100505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:53.100474    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"]
Oct 30 06:08:53.129074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:53.129044    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"]
Oct 30 06:08:53.256108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:53.256079    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:53.400656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:53.400609    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:53.548409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:53.548378    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:54.066869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:54.066836    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262" podUID="fb360e09-c857-4ffb-ae6f-c685c24f56d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:54.067064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:54.066837    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k" podUID="8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:08:55.283529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:55.283495    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"]
Oct 30 06:08:55.295381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:55.295327    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:55.312723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:55.312582    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"]
Oct 30 06:08:55.445285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:55.445252    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:55.577357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:55.577266    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:57.047552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:57.047521    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:58.101912 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.101856    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh" podUID="a153bad2-aa94-4c96-b259-e487efedb6d0" containerName="payload-processing" containerID="cri-o://021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4" gracePeriod=25
Oct 30 06:08:58.134741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.129970    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz" podUID="54f1bc22-d6ba-479f-9005-a6a90d51e882" containerName="payload-pre-processing" containerID="cri-o://790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247" gracePeriod=25
Oct 30 06:08:58.767730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.767692    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:08:58.844261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.844217    5250 generic.go:356] "Generic (PLEG): container finished" podID="a153bad2-aa94-4c96-b259-e487efedb6d0" containerID="021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4" exitCode=0
Oct 30 06:08:58.844508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.844483    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh" event={"ID":"a153bad2-aa94-4c96-b259-e487efedb6d0","Type":"ContainerDied","Data":"021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4"}
Oct 30 06:08:58.844604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.844586    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh" event={"ID":"a153bad2-aa94-4c96-b259-e487efedb6d0","Type":"ContainerDied","Data":"cb58bd9a61f093d7b999b5200be00b7c809af25b968ee3dccceec8532a90b67f"}
Oct 30 06:08:58.844670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.844625    5250 scope.go:122] "RemoveContainer" containerID="021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4"
Oct 30 06:08:58.844815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.844798    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"
Oct 30 06:08:58.880838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.880804    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a153bad2-aa94-4c96-b259-e487efedb6d0-kube-api-access-8nmh7\" (UniqueName: \"kubernetes.io/projected/a153bad2-aa94-4c96-b259-e487efedb6d0-kube-api-access-8nmh7\") pod \"a153bad2-aa94-4c96-b259-e487efedb6d0\" (UID: \"a153bad2-aa94-4c96-b259-e487efedb6d0\") "
Oct 30 06:08:58.880977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.880958    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a153bad2-aa94-4c96-b259-e487efedb6d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a153bad2-aa94-4c96-b259-e487efedb6d0-plugins-config-volume\") pod \"a153bad2-aa94-4c96-b259-e487efedb6d0\" (UID: \"a153bad2-aa94-4c96-b259-e487efedb6d0\") "
Oct 30 06:08:58.884679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.884645    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a153bad2-aa94-4c96-b259-e487efedb6d0-plugins-config-volume" pod "a153bad2-aa94-4c96-b259-e487efedb6d0" (UID: "a153bad2-aa94-4c96-b259-e487efedb6d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:08:58.885002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.884984    5250 scope.go:122] "RemoveContainer" containerID="021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4"
Oct 30 06:08:58.885654 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:08:58.885575    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4\": container with ID starting with 021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4 not found: ID does not exist" containerID="021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4"
Oct 30 06:08:58.885654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.885615    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4"} err="failed to get container status \"021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4\": rpc error: code = NotFound desc = could not find container \"021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4\": container with ID starting with 021af8ea2dccbdf85035d84371684f90355cb69f259a8412c6b44aa668b629c4 not found: ID does not exist"
Oct 30 06:08:58.890446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.890421    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a153bad2-aa94-4c96-b259-e487efedb6d0-kube-api-access-8nmh7" pod "a153bad2-aa94-4c96-b259-e487efedb6d0" (UID: "a153bad2-aa94-4c96-b259-e487efedb6d0"). InnerVolumeSpecName "kube-api-access-8nmh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:08:58.987801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.987767    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmh7\" (UniqueName: \"kubernetes.io/projected/a153bad2-aa94-4c96-b259-e487efedb6d0-kube-api-access-8nmh7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:58.988066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:58.988049    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a153bad2-aa94-4c96-b259-e487efedb6d0-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:08:59.189501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:59.189468    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"]
Oct 30 06:08:59.199978 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:59.199952    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-38cfad-9979fb864-27bxh"]
Oct 30 06:08:59.282649 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:59.282623    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a153bad2-aa94-4c96-b259-e487efedb6d0" path="/var/lib/kubelet/pods/a153bad2-aa94-4c96-b259-e487efedb6d0/volumes"
Oct 30 06:08:59.535801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:59.535725    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:59.681078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:59.681045    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:08:59.941571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:59.941534    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"]
Oct 30 06:08:59.942081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:59.942045    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy" containerID="cri-o://e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22" gracePeriod=30
Oct 30 06:08:59.991864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:08:59.991826    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:00.284007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.283919    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw" podUID="038e903a-f657-46bd-9b2c-04bd4edfd188" containerName="payload-processing" containerID="cri-o://e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14" gracePeriod=25
Oct 30 06:09:00.313011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.312980    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w" podUID="3dcc57d0-a465-4101-94af-ebed0541ca37" containerName="payload-pre-processing" containerID="cri-o://dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8" gracePeriod=25
Oct 30 06:09:00.579415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.579391    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:09:00.734620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.734589    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038e903a-f657-46bd-9b2c-04bd4edfd188-kube-api-access-fgh5r\" (UniqueName: \"kubernetes.io/projected/038e903a-f657-46bd-9b2c-04bd4edfd188-kube-api-access-fgh5r\") pod \"038e903a-f657-46bd-9b2c-04bd4edfd188\" (UID: \"038e903a-f657-46bd-9b2c-04bd4edfd188\") "
Oct 30 06:09:00.734876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.734854    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038e903a-f657-46bd-9b2c-04bd4edfd188-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038e903a-f657-46bd-9b2c-04bd4edfd188-plugins-config-volume\") pod \"038e903a-f657-46bd-9b2c-04bd4edfd188\" (UID: \"038e903a-f657-46bd-9b2c-04bd4edfd188\") "
Oct 30 06:09:00.735244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.735221    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038e903a-f657-46bd-9b2c-04bd4edfd188-plugins-config-volume" pod "038e903a-f657-46bd-9b2c-04bd4edfd188" (UID: "038e903a-f657-46bd-9b2c-04bd4edfd188"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:00.735419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.735405    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038e903a-f657-46bd-9b2c-04bd4edfd188-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:00.737094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.737061    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038e903a-f657-46bd-9b2c-04bd4edfd188-kube-api-access-fgh5r" pod "038e903a-f657-46bd-9b2c-04bd4edfd188" (UID: "038e903a-f657-46bd-9b2c-04bd4edfd188"). InnerVolumeSpecName "kube-api-access-fgh5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:00.813695 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.813611    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:00.836715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.836685    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgh5r\" (UniqueName: \"kubernetes.io/projected/038e903a-f657-46bd-9b2c-04bd4edfd188-kube-api-access-fgh5r\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:00.865233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.865158    5250 generic.go:356] "Generic (PLEG): container finished" podID="038e903a-f657-46bd-9b2c-04bd4edfd188" containerID="e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14" exitCode=0
Oct 30 06:09:00.865233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.865223    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"
Oct 30 06:09:00.865445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.865254    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw" event={"ID":"038e903a-f657-46bd-9b2c-04bd4edfd188","Type":"ContainerDied","Data":"e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14"}
Oct 30 06:09:00.865445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.865325    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw" event={"ID":"038e903a-f657-46bd-9b2c-04bd4edfd188","Type":"ContainerDied","Data":"dc3c606c0ebdfba3f1e6ee2db766970165c838e6013c717be8bc54983dc913a2"}
Oct 30 06:09:00.865445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.865357    5250 scope.go:122] "RemoveContainer" containerID="e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14"
Oct 30 06:09:00.878429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.878410    5250 scope.go:122] "RemoveContainer" containerID="e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14"
Oct 30 06:09:00.878712 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:00.878691    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14\": container with ID starting with e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14 not found: ID does not exist" containerID="e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14"
Oct 30 06:09:00.878792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.878718    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14"} err="failed to get container status \"e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14\": rpc error: code = NotFound desc = could not find container \"e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14\": container with ID starting with e017964f616aeab7fb99bcebb35bf500cc9cd882189f4921531ec043d8e21a14 not found: ID does not exist"
Oct 30 06:09:00.913612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.913585    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"]
Oct 30 06:09:00.923642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:00.923612    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f28e2d-7dd6f47c86-8hrtw"]
Oct 30 06:09:01.284113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:01.284081    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038e903a-f657-46bd-9b2c-04bd4edfd188" path="/var/lib/kubelet/pods/038e903a-f657-46bd-9b2c-04bd4edfd188/volumes"
Oct 30 06:09:01.291905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:01.291872    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:01.444253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:01.444167    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:01.679300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:01.679267    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:01.995015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:01.994924    5250 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:09:01.995015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:01.994988    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:09:02.375994 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:02.375905    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:09:03.491888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:03.491842    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:03.834217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:03.834127    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"]
Oct 30 06:09:03.834398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:03.834370    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy" containerID="cri-o://208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd" gracePeriod=30
Oct 30 06:09:04.066609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:04.066565    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k" podUID="8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:04.066793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:04.066576    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262" podUID="fb360e09-c857-4ffb-ae6f-c685c24f56d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:04.196359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:04.196328    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz" podUID="54f1bc22-d6ba-479f-9005-a6a90d51e882" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:04.638072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:04.638033    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w" podUID="3dcc57d0-a465-4101-94af-ebed0541ca37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:05.230093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.230059    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:09:05.398047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.398002    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:05.424342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.424289    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea392143-910f-4440-961a-508594fd5d6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea392143-910f-4440-961a-508594fd5d6f-istio-podinfo\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.424728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.424691    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-certs\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.424864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.424762    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-istio-token\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.424864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.424809    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-data\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.426057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.424844    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-credential-socket\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.426057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.424982    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea392143-910f-4440-961a-508594fd5d6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea392143-910f-4440-961a-508594fd5d6f-istiod-ca-cert\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.426057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.425223    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-kube-api-access-5kjzg\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-kube-api-access-5kjzg\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.426057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.425339    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-socket\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.426057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.425441    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-envoy\") pod \"ea392143-910f-4440-961a-508594fd5d6f\" (UID: \"ea392143-910f-4440-961a-508594fd5d6f\") "
Oct 30 06:09:05.427560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.427533    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-credential-socket" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:05.427846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.427812    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-data" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:05.428339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.428308    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-socket" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:05.428733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.428681    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-certs" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:05.435309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.429093    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea392143-910f-4440-961a-508594fd5d6f-istiod-ca-cert" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:05.435309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.432908    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-kube-api-access-5kjzg" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "kube-api-access-5kjzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:05.435309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.432914    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea392143-910f-4440-961a-508594fd5d6f-istio-podinfo" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:09:05.435309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.432977    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-envoy" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:05.436373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.436346    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-istio-token" pod "ea392143-910f-4440-961a-508594fd5d6f" (UID: "ea392143-910f-4440-961a-508594fd5d6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:05.528195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528111    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.528195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528137    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea392143-910f-4440-961a-508594fd5d6f-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.528195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528146    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.528195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528155    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.528195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528166    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.528492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528220    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.528492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528236    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea392143-910f-4440-961a-508594fd5d6f-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.528492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528252    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjzg\" (UniqueName: \"kubernetes.io/projected/ea392143-910f-4440-961a-508594fd5d6f-kube-api-access-5kjzg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.528492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.528263    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea392143-910f-4440-961a-508594fd5d6f-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:05.842075 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.841962    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:05.919708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.919668    5250 generic.go:356] "Generic (PLEG): container finished" podID="ea392143-910f-4440-961a-508594fd5d6f" containerID="e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22" exitCode=0
Oct 30 06:09:05.919892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.919793    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" event={"ID":"ea392143-910f-4440-961a-508594fd5d6f","Type":"ContainerDied","Data":"e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22"}
Oct 30 06:09:05.919892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.919886    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5" event={"ID":"ea392143-910f-4440-961a-508594fd5d6f","Type":"ContainerDied","Data":"03fe2922129cfab99bf058ba0c883e6953cceae1715e1c40abd0514f8b08c40c"}
Oct 30 06:09:05.920025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.919928    5250 scope.go:122] "RemoveContainer" containerID="e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22"
Oct 30 06:09:05.920025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.919991    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"
Oct 30 06:09:05.933729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.933704    5250 scope.go:122] "RemoveContainer" containerID="e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22"
Oct 30 06:09:05.934032 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:05.934008    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22\": container with ID starting with e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22 not found: ID does not exist" containerID="e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22"
Oct 30 06:09:05.934131 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:05.934041    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22"} err="failed to get container status \"e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22\": rpc error: code = NotFound desc = could not find container \"e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22\": container with ID starting with e1c9954af2ae7ee0714975a3d565ad436d726d3a661050626b4bf36f57310a22 not found: ID does not exist"
Oct 30 06:09:06.016907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:06.016873    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"]
Oct 30 06:09:06.045692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:06.045659    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-vxzd5"]
Oct 30 06:09:06.085943 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:06.085915    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:06.302892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:06.302861    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"]
Oct 30 06:09:06.303092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:06.303071    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy" containerID="cri-o://dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92" gracePeriod=30
Oct 30 06:09:06.524412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:06.524381    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"]
Oct 30 06:09:06.524613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:06.524572    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" containerID="cri-o://147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad" gracePeriod=30
Oct 30 06:09:07.283026 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.282991    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea392143-910f-4440-961a-508594fd5d6f" path="/var/lib/kubelet/pods/ea392143-910f-4440-961a-508594fd5d6f/volumes"
Oct 30 06:09:07.480098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.480068    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:07.623786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.623700    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"]
Oct 30 06:09:07.624612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624590    5250 cpu_manager.go:414] "containerMap: removing container" podUID="32b57a2f-7a55-4765-8dcb-52b393877d13" containerName="revoke-keys"
Oct 30 06:09:07.624612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624610    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b57a2f-7a55-4765-8dcb-52b393877d13" containerName="revoke-keys"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624638    5250 cpu_manager.go:414] "containerMap: removing container" podUID="a153bad2-aa94-4c96-b259-e487efedb6d0" containerName="payload-processing"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624646    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a153bad2-aa94-4c96-b259-e487efedb6d0" containerName="payload-processing"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624655    5250 cpu_manager.go:414] "containerMap: removing container" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624662    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624696    5250 cpu_manager.go:414] "containerMap: removing container" podUID="fabeb868-afa6-49c3-b089-5905b37f48e4" containerName="main"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624701    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabeb868-afa6-49c3-b089-5905b37f48e4" containerName="main"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624715    5250 cpu_manager.go:414] "containerMap: removing container" podUID="038e903a-f657-46bd-9b2c-04bd4edfd188" containerName="payload-processing"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624720    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038e903a-f657-46bd-9b2c-04bd4edfd188" containerName="payload-processing"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624740    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy"
Oct 30 06:09:07.624786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624745    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy"
Oct 30 06:09:07.625352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624854    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="a153bad2-aa94-4c96-b259-e487efedb6d0" containerName="payload-processing"
Oct 30 06:09:07.625352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624864    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fd3513-4813-4402-b48c-8a5a761c45c9" containerName="istio-proxy"
Oct 30 06:09:07.625352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624873    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabeb868-afa6-49c3-b089-5905b37f48e4" containerName="main"
Oct 30 06:09:07.625352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624880    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="038e903a-f657-46bd-9b2c-04bd4edfd188" containerName="payload-processing"
Oct 30 06:09:07.625352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624887    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea392143-910f-4440-961a-508594fd5d6f" containerName="istio-proxy"
Oct 30 06:09:07.625352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.624893    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b57a2f-7a55-4765-8dcb-52b393877d13" containerName="revoke-keys"
Oct 30 06:09:07.625563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.625390    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:07.630034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.630018    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:07.648670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.648645    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"]
Oct 30 06:09:07.653791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.653616    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-c2be21\""
Oct 30 06:09:07.653791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.653628    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-c2be21-dockercfg-vb4jd\""
Oct 30 06:09:07.662715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.656473    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj\" (UID: \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:07.662715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.656615    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jm58\" (UniqueName: \"kubernetes.io/projected/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-kube-api-access-9jm58\") pod \"payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj\" (UID: \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:07.707204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.706341    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"]
Oct 30 06:09:07.778201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.777352    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj\" (UID: \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:07.778201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.777554    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jm58\" (UniqueName: \"kubernetes.io/projected/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-kube-api-access-9jm58\") pod \"payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj\" (UID: \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:07.789153 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.789116    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj\" (UID: \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:07.806261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.803426    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:07.844451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.844390    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"]
Oct 30 06:09:07.853196 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.852839    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jm58\" (UniqueName: \"kubernetes.io/projected/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-kube-api-access-9jm58\") pod \"payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj\" (UID: \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:07.945394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.945362    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:07.979954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.979920    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs9v7\" (UniqueName: \"kubernetes.io/projected/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-kube-api-access-zs9v7\") pod \"payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5\" (UID: \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:07.980300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:07.980268    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5\" (UID: \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:08.081656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.081610    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5\" (UID: \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:08.081825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.081779    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs9v7\" (UniqueName: \"kubernetes.io/projected/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-kube-api-access-zs9v7\") pod \"payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5\" (UID: \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:08.082505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.082480    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5\" (UID: \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:08.102572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.102539    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:09:08.102733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.102597    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:09:08.104558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.104533    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs9v7\" (UniqueName: \"kubernetes.io/projected/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-kube-api-access-zs9v7\") pod \"payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5\" (UID: \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:08.157726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.157695    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:08.254944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.254913    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"]
Oct 30 06:09:08.255663 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:09:08.255635    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56bf06b_54bc_4ba4_811b_111b4b02bd0c.slice/crio-99f69c8e295be039a8eaacecd51d81ea90d7d9791dd38713a0b760cf5ea5c773 WatchSource:0}: Error finding container 99f69c8e295be039a8eaacecd51d81ea90d7d9791dd38713a0b760cf5ea5c773: Status 404 returned error can't find the container with id 99f69c8e295be039a8eaacecd51d81ea90d7d9791dd38713a0b760cf5ea5c773
Oct 30 06:09:08.469439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.469407    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"]
Oct 30 06:09:08.471136 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:09:08.471111    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85a1613_f0cb_4107_9ce4_3ab7b2b220c6.slice/crio-4792310eca80fe3f8b3d085ec3787076f3d26bf481c25128842270531abc44fd WatchSource:0}: Error finding container 4792310eca80fe3f8b3d085ec3787076f3d26bf481c25128842270531abc44fd: Status 404 returned error can't find the container with id 4792310eca80fe3f8b3d085ec3787076f3d26bf481c25128842270531abc44fd
Oct 30 06:09:08.659321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.659292    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 06:09:08.713620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.712583    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-plugins-config-volume\") pod \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\" (UID: \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\") "
Oct 30 06:09:08.713620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.712800    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-kube-api-access-hhtg4\" (UniqueName: \"kubernetes.io/projected/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-kube-api-access-hhtg4\") pod \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\" (UID: \"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5\") "
Oct 30 06:09:08.713620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.713150    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-plugins-config-volume" pod "8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" (UID: "8ea91ab0-130d-4bc0-93a9-41a79bfebbe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:08.713620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.713431    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:08.716092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.716067    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-kube-api-access-hhtg4" pod "8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" (UID: "8ea91ab0-130d-4bc0-93a9-41a79bfebbe5"). InnerVolumeSpecName "kube-api-access-hhtg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:08.725538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.725484    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 06:09:08.814270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.814236    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb360e09-c857-4ffb-ae6f-c685c24f56d1-kube-api-access-k656l\" (UniqueName: \"kubernetes.io/projected/fb360e09-c857-4ffb-ae6f-c685c24f56d1-kube-api-access-k656l\") pod \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\" (UID: \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\") "
Oct 30 06:09:08.814501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.814411    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb360e09-c857-4ffb-ae6f-c685c24f56d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb360e09-c857-4ffb-ae6f-c685c24f56d1-plugins-config-volume\") pod \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\" (UID: \"fb360e09-c857-4ffb-ae6f-c685c24f56d1\") "
Oct 30 06:09:08.814849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.814824    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb360e09-c857-4ffb-ae6f-c685c24f56d1-plugins-config-volume" pod "fb360e09-c857-4ffb-ae6f-c685c24f56d1" (UID: "fb360e09-c857-4ffb-ae6f-c685c24f56d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:08.815047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.815033    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhtg4\" (UniqueName: \"kubernetes.io/projected/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5-kube-api-access-hhtg4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:08.815094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.815051    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb360e09-c857-4ffb-ae6f-c685c24f56d1-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:08.816578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.816554    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb360e09-c857-4ffb-ae6f-c685c24f56d1-kube-api-access-k656l" pod "fb360e09-c857-4ffb-ae6f-c685c24f56d1" (UID: "fb360e09-c857-4ffb-ae6f-c685c24f56d1"). InnerVolumeSpecName "kube-api-access-k656l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:08.916596 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.916560    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k656l\" (UniqueName: \"kubernetes.io/projected/fb360e09-c857-4ffb-ae6f-c685c24f56d1-kube-api-access-k656l\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:08.950867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.950841    5250 generic.go:356] "Generic (PLEG): container finished" podID="fb360e09-c857-4ffb-ae6f-c685c24f56d1" containerID="e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf" exitCode=0
Oct 30 06:09:08.950994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.950881    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"
Oct 30 06:09:08.950994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.950927    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262" event={"ID":"fb360e09-c857-4ffb-ae6f-c685c24f56d1","Type":"ContainerDied","Data":"e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf"}
Oct 30 06:09:08.950994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.950985    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262" event={"ID":"fb360e09-c857-4ffb-ae6f-c685c24f56d1","Type":"ContainerDied","Data":"3668fa09470e31fc86146608752ff61860e671980e4905878e3fe5441a42d930"}
Oct 30 06:09:08.951215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.951015    5250 scope.go:122] "RemoveContainer" containerID="e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf"
Oct 30 06:09:08.953219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.953160    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj" event={"ID":"b56bf06b-54bc-4ba4-811b-111b4b02bd0c","Type":"ContainerStarted","Data":"66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5"}
Oct 30 06:09:08.953348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.953250    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:08.953348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.953274    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj" event={"ID":"b56bf06b-54bc-4ba4-811b-111b4b02bd0c","Type":"ContainerStarted","Data":"99f69c8e295be039a8eaacecd51d81ea90d7d9791dd38713a0b760cf5ea5c773"}
Oct 30 06:09:08.954680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.954662    5250 generic.go:356] "Generic (PLEG): container finished" podID="8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" containerID="4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136" exitCode=0
Oct 30 06:09:08.954769 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.954706    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"
Oct 30 06:09:08.954769 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.954756    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k" event={"ID":"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5","Type":"ContainerDied","Data":"4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136"}
Oct 30 06:09:08.954897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.954807    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k" event={"ID":"8ea91ab0-130d-4bc0-93a9-41a79bfebbe5","Type":"ContainerDied","Data":"3909b501934372e25408d443e60b00122ecae4909556434ade0af9df162c390e"}
Oct 30 06:09:08.963831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.963799    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5" event={"ID":"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6","Type":"ContainerStarted","Data":"2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b"}
Oct 30 06:09:08.963939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.963902    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5" event={"ID":"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6","Type":"ContainerStarted","Data":"4792310eca80fe3f8b3d085ec3787076f3d26bf481c25128842270531abc44fd"}
Oct 30 06:09:08.964790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.964769    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:08.971835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.971728    5250 scope.go:122] "RemoveContainer" containerID="e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf"
Oct 30 06:09:08.972232 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:08.972206    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf\": container with ID starting with e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf not found: ID does not exist" containerID="e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf"
Oct 30 06:09:08.972385 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.972248    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf"} err="failed to get container status \"e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf\": rpc error: code = NotFound desc = could not find container \"e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf\": container with ID starting with e210599b07a7fa4db6a084e8ed3a2c80d1adebca864df4eb059480567c51abcf not found: ID does not exist"
Oct 30 06:09:08.972385 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.972270    5250 scope.go:122] "RemoveContainer" containerID="4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136"
Oct 30 06:09:08.988985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.988960    5250 scope.go:122] "RemoveContainer" containerID="4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136"
Oct 30 06:09:08.989379 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:08.989352    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136\": container with ID starting with 4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136 not found: ID does not exist" containerID="4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136"
Oct 30 06:09:08.989497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:08.989388    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136"} err="failed to get container status \"4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136\": rpc error: code = NotFound desc = could not find container \"4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136\": container with ID starting with 4518aaed5314ab9fc7329c7b291b205f385b4545874c7918c1119aca549fd136 not found: ID does not exist"
Oct 30 06:09:09.006482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.006421    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj" podStartSLOduration=2.0064063069999998 podStartE2EDuration="2.006406307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:09.002835634 +0000 UTC m=+2263.833872852" watchObservedRunningTime="2026-10-30 06:09:09.006406307 +0000 UTC m=+2263.837443525"
Oct 30 06:09:09.043485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.043436    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5" podStartSLOduration=2.043422553 podStartE2EDuration="2.043422553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:09.040397975 +0000 UTC m=+2263.871435193" watchObservedRunningTime="2026-10-30 06:09:09.043422553 +0000 UTC m=+2263.874459767"
Oct 30 06:09:09.089080 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.089046    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"]
Oct 30 06:09:09.093612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.093587    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-482468-8589b6c84-pgp6k"]
Oct 30 06:09:09.122069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.122035    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"]
Oct 30 06:09:09.126825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.126803    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-482468-698df57b87-zg262"]
Oct 30 06:09:09.127990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.127970    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 06:09:09.220534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.220488    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c88e80-0349-4523-bacc-048e69785ab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c88e80-0349-4523-bacc-048e69785ab9-istio-podinfo\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.220733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.220594    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-envoy\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.220733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.220691    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c88e80-0349-4523-bacc-048e69785ab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c88e80-0349-4523-bacc-048e69785ab9-istiod-ca-cert\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.220871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.220735    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-credential-socket\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.220871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.220807    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-socket\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.220977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.220883    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-certs\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.220977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.220967    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-istio-token\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.221089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.220995    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-kube-api-access-f449w\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-kube-api-access-f449w\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.221089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.221083    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-data\") pod \"f9c88e80-0349-4523-bacc-048e69785ab9\" (UID: \"f9c88e80-0349-4523-bacc-048e69785ab9\") "
Oct 30 06:09:09.221347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.221321    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c88e80-0349-4523-bacc-048e69785ab9-istiod-ca-cert" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:09.221729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.221705    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c88e80-0349-4523-bacc-048e69785ab9-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.222359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.222094    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-certs" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:09.222486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.222352    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-socket" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:09.222486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.222195    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-credential-socket" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:09.222486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.222417    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-data" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:09.224307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.224271    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-envoy" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:09.224631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.224607    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c88e80-0349-4523-bacc-048e69785ab9-istio-podinfo" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:09:09.224974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.224943    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-istio-token" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:09.225683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.225652    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-kube-api-access-f449w" pod "f9c88e80-0349-4523-bacc-048e69785ab9" (UID: "f9c88e80-0349-4523-bacc-048e69785ab9"). InnerVolumeSpecName "kube-api-access-f449w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:09.283081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.283010    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" path="/var/lib/kubelet/pods/8ea91ab0-130d-4bc0-93a9-41a79bfebbe5/volumes"
Oct 30 06:09:09.283542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.283526    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb360e09-c857-4ffb-ae6f-c685c24f56d1" path="/var/lib/kubelet/pods/fb360e09-c857-4ffb-ae6f-c685c24f56d1/volumes"
Oct 30 06:09:09.296322 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.296300    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:09.323757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.323721    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.323757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.323755    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.323965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.323766    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.323965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.323777    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.323965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.323789    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f449w\" (UniqueName: \"kubernetes.io/projected/f9c88e80-0349-4523-bacc-048e69785ab9-kube-api-access-f449w\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.323965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.323801    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.323965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.323809    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c88e80-0349-4523-bacc-048e69785ab9-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.323965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.323816    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c88e80-0349-4523-bacc-048e69785ab9-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:09.973111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.973078    5250 generic.go:356] "Generic (PLEG): container finished" podID="f9c88e80-0349-4523-bacc-048e69785ab9" containerID="208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd" exitCode=0
Oct 30 06:09:09.973608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.973124    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"
Oct 30 06:09:09.973608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.973155    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" event={"ID":"f9c88e80-0349-4523-bacc-048e69785ab9","Type":"ContainerDied","Data":"208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd"}
Oct 30 06:09:09.973608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.973241    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94" event={"ID":"f9c88e80-0349-4523-bacc-048e69785ab9","Type":"ContainerDied","Data":"d71d739da3ea5985505f9df6698ac643549dc2c767c8cd9bf35686c749b05f4a"}
Oct 30 06:09:09.973608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.973272    5250 scope.go:122] "RemoveContainer" containerID="208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd"
Oct 30 06:09:09.984825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.984809    5250 scope.go:122] "RemoveContainer" containerID="208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd"
Oct 30 06:09:09.985111 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:09.985085    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd\": container with ID starting with 208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd not found: ID does not exist" containerID="208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd"
Oct 30 06:09:09.985233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:09.985119    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd"} err="failed to get container status \"208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd\": rpc error: code = NotFound desc = could not find container \"208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd\": container with ID starting with 208c481a50d0276c85a83e4865022cd9bc321743f1bf50083e27b83c256886bd not found: ID does not exist"
Oct 30 06:09:10.022384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:10.022359    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"]
Oct 30 06:09:10.035228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:10.035205    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-hlw94"]
Oct 30 06:09:10.237596 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:10.237519    5250 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:09:10.237596 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:10.237578    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:09:10.729021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:10.728988    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:11.240539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.240501    5250 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:09:11.240924 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.240570    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:09:11.282915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.282881    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" path="/var/lib/kubelet/pods/f9c88e80-0349-4523-bacc-048e69785ab9/volumes"
Oct 30 06:09:11.376778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.376740    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:11.595141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.595114    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:11.667174 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.667134    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 06:09:11.760478 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.760404    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:11.765507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765484    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-certs\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.765665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765517    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-envoy\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.765665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765641    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-podinfo\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.765791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765696    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istiod-ca-cert\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.765791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765735    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-socket\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.765901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765820    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-kube-api-access-mwt49\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-kube-api-access-mwt49\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.765954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765922    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-token\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-token\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.766004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765962    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-data\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.766072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.766053    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-credential-socket\") pod \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\" (UID: \"ef239f1c-5e8d-4450-aedd-ca14c332dd36\") "
Oct 30 06:09:11.766154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.765812    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-certs" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.766249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.766118    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istiod-ca-cert" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:11.766532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.766514    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-data" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.766750 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.766725    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.766931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.766756    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.766931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.766770    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.767072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.766972    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-socket" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.767157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.767137    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-credential-socket" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.769641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.769615    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-kube-api-access-mwt49" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "kube-api-access-mwt49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:11.769755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.769697    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-envoy" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.769860 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.769834    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-token" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:11.770866 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.770846    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-podinfo" pod "ef239f1c-5e8d-4450-aedd-ca14c332dd36" (UID: "ef239f1c-5e8d-4450-aedd-ca14c332dd36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:09:11.803971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.803948    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 06:09:11.868822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868230    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-credential-socket\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.868822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868307    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-data\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.868822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868366    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-envoy\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.868822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868401    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d887fd9-40b4-44c7-8c81-89a824d26ace-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d887fd9-40b4-44c7-8c81-89a824d26ace-istiod-ca-cert\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.868822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868482    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-kube-api-access-ln2zf\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-kube-api-access-ln2zf\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.868822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868725    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-certs\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.868822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868819    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-socket\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.869326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868867    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-token\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-token\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.869326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.868902    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-podinfo\") pod \"8d887fd9-40b4-44c7-8c81-89a824d26ace\" (UID: \"8d887fd9-40b4-44c7-8c81-89a824d26ace\") "
Oct 30 06:09:11.869619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.869466    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-credential-socket" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.869715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.869694    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-data" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.869894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.869852    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d887fd9-40b4-44c7-8c81-89a824d26ace-istiod-ca-cert" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:11.870035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.869973    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-socket" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.870035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.870012    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.870035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.870033    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwt49\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-kube-api-access-mwt49\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.870272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.870065    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.870272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.870080    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.870272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.870093    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.870272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.870109    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef239f1c-5e8d-4450-aedd-ca14c332dd36-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.870272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.870230    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-certs" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.875751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.875727    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-podinfo" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:09:11.876983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.876945    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-token" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:11.877405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.877382    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-envoy" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:11.879352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.879330    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-kube-api-access-ln2zf" pod "8d887fd9-40b4-44c7-8c81-89a824d26ace" (UID: "8d887fd9-40b4-44c7-8c81-89a824d26ace"). InnerVolumeSpecName "kube-api-access-ln2zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:11.971355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971324    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.971355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971351    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.971355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971360    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.971589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971368    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.971589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971378    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.971589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971386    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.971589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971394    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d887fd9-40b4-44c7-8c81-89a824d26ace-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.971589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971403    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln2zf\" (UniqueName: \"kubernetes.io/projected/8d887fd9-40b4-44c7-8c81-89a824d26ace-kube-api-access-ln2zf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.971589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.971411    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d887fd9-40b4-44c7-8c81-89a824d26ace-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:11.999345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.999313    5250 generic.go:356] "Generic (PLEG): container finished" podID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerID="147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad" exitCode=0
Oct 30 06:09:11.999499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.999366    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"
Oct 30 06:09:11.999499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.999366    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" event={"ID":"8d887fd9-40b4-44c7-8c81-89a824d26ace","Type":"ContainerDied","Data":"147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad"}
Oct 30 06:09:11.999499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.999434    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj" event={"ID":"8d887fd9-40b4-44c7-8c81-89a824d26ace","Type":"ContainerDied","Data":"a5b21b8e28e46d749afbb4070373f84cb09910353553350f17ffa3f2c7485899"}
Oct 30 06:09:11.999499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:11.999463    5250 scope.go:122] "RemoveContainer" containerID="147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad"
Oct 30 06:09:12.001190 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.001154    5250 generic.go:356] "Generic (PLEG): container finished" podID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerID="dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92" exitCode=0
Oct 30 06:09:12.001312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.001232    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"
Oct 30 06:09:12.001312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.001243    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" event={"ID":"ef239f1c-5e8d-4450-aedd-ca14c332dd36","Type":"ContainerDied","Data":"dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92"}
Oct 30 06:09:12.001312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.001305    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w" event={"ID":"ef239f1c-5e8d-4450-aedd-ca14c332dd36","Type":"ContainerDied","Data":"31ec5e93b5653173351965245e24db2d824cfc2bafad78edcefe7214c8749bdc"}
Oct 30 06:09:12.012654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.012631    5250 scope.go:122] "RemoveContainer" containerID="147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad"
Oct 30 06:09:12.012966 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:12.012947    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad\": container with ID starting with 147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad not found: ID does not exist" containerID="147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad"
Oct 30 06:09:12.013052 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.012975    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad"} err="failed to get container status \"147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad\": rpc error: code = NotFound desc = could not find container \"147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad\": container with ID starting with 147cc42cab090c2fcf484e6ed2766dc3dd75d293ddb04044bf55e70d5bcb5bad not found: ID does not exist"
Oct 30 06:09:12.013052 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.013000    5250 scope.go:122] "RemoveContainer" containerID="dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92"
Oct 30 06:09:12.023265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.023249    5250 scope.go:122] "RemoveContainer" containerID="dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92"
Oct 30 06:09:12.023530 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:12.023509    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92\": container with ID starting with dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92 not found: ID does not exist" containerID="dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92"
Oct 30 06:09:12.023595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.023538    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92"} err="failed to get container status \"dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92\": rpc error: code = NotFound desc = could not find container \"dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92\": container with ID starting with dbc053bcdee1a314cc61996bd46ed8570ca7c0c7cfb2322ee0840b90bed99c92 not found: ID does not exist"
Oct 30 06:09:12.050326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.050220    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"]
Oct 30 06:09:12.062229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.061454    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-n8bkj"]
Oct 30 06:09:12.094380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.094348    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"]
Oct 30 06:09:12.098541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.098519    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-7l84w"]
Oct 30 06:09:12.142970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:12.142930    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:13.283037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:13.283007    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" path="/var/lib/kubelet/pods/8d887fd9-40b4-44c7-8c81-89a824d26ace/volumes"
Oct 30 06:09:13.283546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:13.283530    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" path="/var/lib/kubelet/pods/ef239f1c-5e8d-4450-aedd-ca14c332dd36/volumes"
Oct 30 06:09:13.723623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:13.723572    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:14.196077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:14.196033    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz" podUID="54f1bc22-d6ba-479f-9005-a6a90d51e882" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:14.638264 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:14.638225    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w" podUID="3dcc57d0-a465-4101-94af-ebed0541ca37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:15.469273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:15.469237    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:17.298935 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:17.298855    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:18.281402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.281370    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:18.428419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.428390    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:18.471243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.471221    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:09:18.562693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.562655    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f1bc22-d6ba-479f-9005-a6a90d51e882-kube-api-access-m2zln\" (UniqueName: \"kubernetes.io/projected/54f1bc22-d6ba-479f-9005-a6a90d51e882-kube-api-access-m2zln\") pod \"54f1bc22-d6ba-479f-9005-a6a90d51e882\" (UID: \"54f1bc22-d6ba-479f-9005-a6a90d51e882\") "
Oct 30 06:09:18.562928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.562899    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f1bc22-d6ba-479f-9005-a6a90d51e882-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f1bc22-d6ba-479f-9005-a6a90d51e882-plugins-config-volume\") pod \"54f1bc22-d6ba-479f-9005-a6a90d51e882\" (UID: \"54f1bc22-d6ba-479f-9005-a6a90d51e882\") "
Oct 30 06:09:18.571234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.567197    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f1bc22-d6ba-479f-9005-a6a90d51e882-plugins-config-volume" pod "54f1bc22-d6ba-479f-9005-a6a90d51e882" (UID: "54f1bc22-d6ba-479f-9005-a6a90d51e882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:18.572514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.572480    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f1bc22-d6ba-479f-9005-a6a90d51e882-kube-api-access-m2zln" pod "54f1bc22-d6ba-479f-9005-a6a90d51e882" (UID: "54f1bc22-d6ba-479f-9005-a6a90d51e882"). InnerVolumeSpecName "kube-api-access-m2zln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:18.592683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.592567    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"]
Oct 30 06:09:18.592856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.592829    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy" containerID="cri-o://2d1af394e4e45c5c8adbe9d70061c41c0d294d4b4bf4e2c881d1b54b19a6196d" gracePeriod=30
Oct 30 06:09:18.659021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.658991    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:18.664741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.664714    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2zln\" (UniqueName: \"kubernetes.io/projected/54f1bc22-d6ba-479f-9005-a6a90d51e882-kube-api-access-m2zln\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:18.664741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:18.664740    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f1bc22-d6ba-479f-9005-a6a90d51e882-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:19.064867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.064833    5250 generic.go:356] "Generic (PLEG): container finished" podID="54f1bc22-d6ba-479f-9005-a6a90d51e882" containerID="790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247" exitCode=0
Oct 30 06:09:19.065083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.064873    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"
Oct 30 06:09:19.065083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.064909    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz" event={"ID":"54f1bc22-d6ba-479f-9005-a6a90d51e882","Type":"ContainerDied","Data":"790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247"}
Oct 30 06:09:19.065083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.064965    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz" event={"ID":"54f1bc22-d6ba-479f-9005-a6a90d51e882","Type":"ContainerDied","Data":"9c3aca6a79cd1f4ccc47d7fdc2920d5612702260e8b617a50fee6350f533dc47"}
Oct 30 06:09:19.065083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.064993    5250 scope.go:122] "RemoveContainer" containerID="790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247"
Oct 30 06:09:19.076922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.076904    5250 scope.go:122] "RemoveContainer" containerID="790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247"
Oct 30 06:09:19.077202 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:19.077166    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247\": container with ID starting with 790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247 not found: ID does not exist" containerID="790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247"
Oct 30 06:09:19.077254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.077213    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247"} err="failed to get container status \"790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247\": rpc error: code = NotFound desc = could not find container \"790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247\": container with ID starting with 790a8e46b72c6e52edaf6ec89d53270476440a91d187b9431ed165b2f1985247 not found: ID does not exist"
Oct 30 06:09:19.113341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.113316    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"]
Oct 30 06:09:19.117410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.117387    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-38cfad-c76fbb5b7-qmpgz"]
Oct 30 06:09:19.189163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.189119    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:19.282340 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.282308    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f1bc22-d6ba-479f-9005-a6a90d51e882" path="/var/lib/kubelet/pods/54f1bc22-d6ba-479f-9005-a6a90d51e882/volumes"
Oct 30 06:09:19.979808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:19.979778    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:09:20.592446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.592424    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:09:20.699154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.699121    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dcc57d0-a465-4101-94af-ebed0541ca37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcc57d0-a465-4101-94af-ebed0541ca37-plugins-config-volume\") pod \"3dcc57d0-a465-4101-94af-ebed0541ca37\" (UID: \"3dcc57d0-a465-4101-94af-ebed0541ca37\") "
Oct 30 06:09:20.699333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.699224    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dcc57d0-a465-4101-94af-ebed0541ca37-kube-api-access-2sld9\" (UniqueName: \"kubernetes.io/projected/3dcc57d0-a465-4101-94af-ebed0541ca37-kube-api-access-2sld9\") pod \"3dcc57d0-a465-4101-94af-ebed0541ca37\" (UID: \"3dcc57d0-a465-4101-94af-ebed0541ca37\") "
Oct 30 06:09:20.699613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.699576    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dcc57d0-a465-4101-94af-ebed0541ca37-plugins-config-volume" pod "3dcc57d0-a465-4101-94af-ebed0541ca37" (UID: "3dcc57d0-a465-4101-94af-ebed0541ca37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:20.699764 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.699749    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcc57d0-a465-4101-94af-ebed0541ca37-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:20.701571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.701550    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcc57d0-a465-4101-94af-ebed0541ca37-kube-api-access-2sld9" pod "3dcc57d0-a465-4101-94af-ebed0541ca37" (UID: "3dcc57d0-a465-4101-94af-ebed0541ca37"). InnerVolumeSpecName "kube-api-access-2sld9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:20.800895 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.800856    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sld9\" (UniqueName: \"kubernetes.io/projected/3dcc57d0-a465-4101-94af-ebed0541ca37-kube-api-access-2sld9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:20.906987 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.906947    5250 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:09:20.907173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.907015    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:09:20.929904 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.929832    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:20.992821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:20.992797    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:09:21.087812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.087778    5250 generic.go:356] "Generic (PLEG): container finished" podID="3dcc57d0-a465-4101-94af-ebed0541ca37" containerID="dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8" exitCode=0
Oct 30 06:09:21.088000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.087832    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"
Oct 30 06:09:21.088000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.087866    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w" event={"ID":"3dcc57d0-a465-4101-94af-ebed0541ca37","Type":"ContainerDied","Data":"dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8"}
Oct 30 06:09:21.088000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.087949    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w" event={"ID":"3dcc57d0-a465-4101-94af-ebed0541ca37","Type":"ContainerDied","Data":"46caee475b75a4096d5d74efc1810adb2f7fa6c7b8b7773fcf62c700120206a9"}
Oct 30 06:09:21.088000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.087982    5250 scope.go:122] "RemoveContainer" containerID="dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8"
Oct 30 06:09:21.101672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.101650    5250 scope.go:122] "RemoveContainer" containerID="dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8"
Oct 30 06:09:21.102037 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:21.101999    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8\": container with ID starting with dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8 not found: ID does not exist" containerID="dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8"
Oct 30 06:09:21.102135 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.102030    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8"} err="failed to get container status \"dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8\": rpc error: code = NotFound desc = could not find container \"dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8\": container with ID starting with dd3db8aec1c3e10da54cd01655a60848b00d798b5bf380184be9bf18b5e167f8 not found: ID does not exist"
Oct 30 06:09:21.145578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.145542    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"]
Oct 30 06:09:21.149362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.149333    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f28e2d-5f499f84f9-7qh9w"]
Oct 30 06:09:21.296668 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.296577    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dcc57d0-a465-4101-94af-ebed0541ca37" path="/var/lib/kubelet/pods/3dcc57d0-a465-4101-94af-ebed0541ca37/volumes"
Oct 30 06:09:21.358250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.358213    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:21.661134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.661093    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:21.823209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:21.823065    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:22.252894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:22.252861    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:22.399497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:22.399449    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:22.461489 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:22.461453    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"]
Oct 30 06:09:22.478439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:22.478403    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-38cfad-tftt2"]
Oct 30 06:09:22.552815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:22.552729    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:22.695698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:22.695663    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:23.001100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:23.001070    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:23.282081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:23.282003    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b57a2f-7a55-4765-8dcb-52b393877d13" path="/var/lib/kubelet/pods/32b57a2f-7a55-4765-8dcb-52b393877d13/volumes"
Oct 30 06:09:24.116243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.116212    5250 generic.go:356] "Generic (PLEG): container finished" podID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerID="2d1af394e4e45c5c8adbe9d70061c41c0d294d4b4bf4e2c881d1b54b19a6196d" exitCode=0
Oct 30 06:09:24.116381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.116291    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" event={"ID":"5f52ebb8-6bcf-4483-b7a1-4f2233315baf","Type":"ContainerDied","Data":"2d1af394e4e45c5c8adbe9d70061c41c0d294d4b4bf4e2c881d1b54b19a6196d"}
Oct 30 06:09:24.272387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.272364    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:09:24.373550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373466    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-kube-api-access-ss6nl\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-kube-api-access-ss6nl\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.373550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373537    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-certs\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373595    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-socket\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373633    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-podinfo\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373675    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istiod-ca-cert\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373752    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-credential-socket\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373917    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-envoy\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373924    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-certs" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373958    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-socket" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.373986    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-token\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-token\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.374090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.374032    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-data\") pod \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\" (UID: \"5f52ebb8-6bcf-4483-b7a1-4f2233315baf\") "
Oct 30 06:09:24.374805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.374133    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-credential-socket" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:24.374805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.374520    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.374805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.374538    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.374805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.374553    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.374805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.374641    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istiod-ca-cert" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:24.375086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.374908    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-data" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:24.376324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.376293    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-kube-api-access-ss6nl" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "kube-api-access-ss6nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:24.376503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.376390    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-envoy" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:24.376691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.376673    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-podinfo" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:09:24.376964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.376942    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-token" pod "5f52ebb8-6bcf-4483-b7a1-4f2233315baf" (UID: "5f52ebb8-6bcf-4483-b7a1-4f2233315baf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:24.466007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.465969    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:24.477219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.476761    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.477219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.476860    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.477219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.476878    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss6nl\" (UniqueName: \"kubernetes.io/projected/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-kube-api-access-ss6nl\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.477219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.476893    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.477219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.476906    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.477219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.476920    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f52ebb8-6bcf-4483-b7a1-4f2233315baf-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:24.737412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:24.737377    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:25.131788 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:25.131698    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l" event={"ID":"5f52ebb8-6bcf-4483-b7a1-4f2233315baf","Type":"ContainerDied","Data":"0412540ff4dae10c718ec19f6b6e28a61454de8f6aac95000cf80cd1167f249b"}
Oct 30 06:09:25.131984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:25.131804    5250 scope.go:122] "RemoveContainer" containerID="2d1af394e4e45c5c8adbe9d70061c41c0d294d4b4bf4e2c881d1b54b19a6196d"
Oct 30 06:09:25.131984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:25.131966    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"
Oct 30 06:09:25.218156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:25.218119    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"]
Oct 30 06:09:25.227859 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:25.227833    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-vwg2l"]
Oct 30 06:09:25.283817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:25.283774    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" path="/var/lib/kubelet/pods/5f52ebb8-6bcf-4483-b7a1-4f2233315baf/volumes"
Oct 30 06:09:26.214155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:26.214122    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:28.246791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:28.246688    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:28.956496 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:28.956449    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:29.099766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.099727    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:29.307593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.307505    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:29.606354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.606274    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"]
Oct 30 06:09:29.639203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.639160    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"]
Oct 30 06:09:29.789926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.789893    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:29.819020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.818986    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"]
Oct 30 06:09:29.819604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819589    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" containerName="payload-processing"
Oct 30 06:09:29.819658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819606    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" containerName="payload-processing"
Oct 30 06:09:29.819658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819622    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy"
Oct 30 06:09:29.819658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819627    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy"
Oct 30 06:09:29.819658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819639    5250 cpu_manager.go:414] "containerMap: removing container" podUID="fb360e09-c857-4ffb-ae6f-c685c24f56d1" containerName="payload-pre-processing"
Oct 30 06:09:29.819658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819646    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb360e09-c857-4ffb-ae6f-c685c24f56d1" containerName="payload-pre-processing"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819667    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819672    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819677    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3dcc57d0-a465-4101-94af-ebed0541ca37" containerName="payload-pre-processing"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819682    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dcc57d0-a465-4101-94af-ebed0541ca37" containerName="payload-pre-processing"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819696    5250 cpu_manager.go:414] "containerMap: removing container" podUID="54f1bc22-d6ba-479f-9005-a6a90d51e882" containerName="payload-pre-processing"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819701    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f1bc22-d6ba-479f-9005-a6a90d51e882" containerName="payload-pre-processing"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819711    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819716    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819736    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy"
Oct 30 06:09:29.819825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819740    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy"
Oct 30 06:09:29.820129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819835    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f1bc22-d6ba-479f-9005-a6a90d51e882" containerName="payload-pre-processing"
Oct 30 06:09:29.820129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819845    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f52ebb8-6bcf-4483-b7a1-4f2233315baf" containerName="istio-proxy"
Oct 30 06:09:29.820129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819855    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef239f1c-5e8d-4450-aedd-ca14c332dd36" containerName="istio-proxy"
Oct 30 06:09:29.820129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819860    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea91ab0-130d-4bc0-93a9-41a79bfebbe5" containerName="payload-processing"
Oct 30 06:09:29.820129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819866    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d887fd9-40b4-44c7-8c81-89a824d26ace" containerName="istio-proxy"
Oct 30 06:09:29.820129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819873    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb360e09-c857-4ffb-ae6f-c685c24f56d1" containerName="payload-pre-processing"
Oct 30 06:09:29.820129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819881    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c88e80-0349-4523-bacc-048e69785ab9" containerName="istio-proxy"
Oct 30 06:09:29.820129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.819887    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dcc57d0-a465-4101-94af-ebed0541ca37" containerName="payload-pre-processing"
Oct 30 06:09:29.825369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.825353    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:29.848384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.848345    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:09:29.848784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.848689    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:09:29.852633 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.852589    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"]
Oct 30 06:09:29.938718 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.938674    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:29.948788 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.948746    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:29.948996 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.948976    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:29.949043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:29.949034    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmh9j\" (UniqueName: \"kubernetes.io/projected/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-kube-api-access-hmh9j\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:30.049822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.049791    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:30.049989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.049919    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:30.049989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.049965    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmh9j\" (UniqueName: \"kubernetes.io/projected/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-kube-api-access-hmh9j\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:30.050661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.050639    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:30.052459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.052432    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:30.072035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.072004    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmh9j\" (UniqueName: \"kubernetes.io/projected/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-kube-api-access-hmh9j\") pod \"maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:30.102870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.102835    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:30.151712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.151683    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:30.329331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.329295    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:30.494394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.494357    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:30.541246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.541220    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"]
Oct 30 06:09:30.542882 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:09:30.542854    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a02a5a8_acf1_47a1_a43e_21d9f90cca9a.slice/crio-6e678bd8714ebeeb60d567eeb2b3d9aaa2ab8879afb9fac959ae5e7b12e8c318 WatchSource:0}: Error finding container 6e678bd8714ebeeb60d567eeb2b3d9aaa2ab8879afb9fac959ae5e7b12e8c318: Status 404 returned error can't find the container with id 6e678bd8714ebeeb60d567eeb2b3d9aaa2ab8879afb9fac959ae5e7b12e8c318
Oct 30 06:09:30.639964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.639928    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"]
Oct 30 06:09:30.640238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.640205    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy" containerID="cri-o://74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02" gracePeriod=30
Oct 30 06:09:30.710678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:30.710641    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:31.187833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:31.187796    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn" event={"ID":"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a","Type":"ContainerStarted","Data":"ea6a460fd2062c564b7c75342ae9c77226d67b0452f2921750fc32a9b23a8928"}
Oct 30 06:09:31.187999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:31.187854    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn" event={"ID":"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a","Type":"ContainerStarted","Data":"6e678bd8714ebeeb60d567eeb2b3d9aaa2ab8879afb9fac959ae5e7b12e8c318"}
Oct 30 06:09:31.226657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:31.226565    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn" podStartSLOduration=2.226553252 podStartE2EDuration="2.226553252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:31.224363905 +0000 UTC m=+2286.055401124" watchObservedRunningTime="2026-10-30 06:09:31.226553252 +0000 UTC m=+2286.057590470"
Oct 30 06:09:32.196810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:32.196776    5250 generic.go:356] "Generic (PLEG): container finished" podID="9a02a5a8-acf1-47a1-a43e-21d9f90cca9a" containerID="ea6a460fd2062c564b7c75342ae9c77226d67b0452f2921750fc32a9b23a8928" exitCode=0
Oct 30 06:09:32.197208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:32.196864    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn" event={"ID":"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a","Type":"ContainerDied","Data":"ea6a460fd2062c564b7c75342ae9c77226d67b0452f2921750fc32a9b23a8928"}
Oct 30 06:09:32.221069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:32.221040    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:32.375074 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:32.375034    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:09:33.030045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.030011    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"]
Oct 30 06:09:33.034589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.034571    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.057775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.057743    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"]
Oct 30 06:09:33.061140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.061115    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-d96a1f\"/\"openshift-service-ca.crt\""
Oct 30 06:09:33.061337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.061319    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-d96a1f\"/\"default-dockercfg-r4q55\""
Oct 30 06:09:33.061676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.061653    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-d96a1f\"/\"kube-root-ca.crt\""
Oct 30 06:09:33.061676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.061667    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-d96a1f\"/\"auth-test-model-d96a1f-kserve-self-signed-certs\""
Oct 30 06:09:33.095734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.095700    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tls-certs\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.095897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.095755    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-dshm\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.095897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.095835    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8v6n\" (UniqueName: \"kubernetes.io/projected/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-kube-api-access-c8v6n\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.096029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.096007    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-model-cache\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.096144 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.096126    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-home\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.096295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.096267    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tmp-dir\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.124090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.124051    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:33.199200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.197709    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tmp-dir\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.199200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.197789    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tls-certs\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.199200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.197844    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-dshm\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.199200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.197905    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8v6n\" (UniqueName: \"kubernetes.io/projected/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-kube-api-access-c8v6n\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.199200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.198227    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-model-cache\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.199200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.198320    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-home\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.199200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.198825    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-home\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.199200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.199152    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-model-cache\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.200094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.199476    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tmp-dir\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.202533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.202506    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tls-certs\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.207482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.207455    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-dshm\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.221009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.220980    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8v6n\" (UniqueName: \"kubernetes.io/projected/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-kube-api-access-c8v6n\") pod \"auth-test-model-d96a1f-kserve-b486749f7-sjwr6\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") " pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.365643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.365359    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:33.450812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.450782    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:33.542858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.542806    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-xks-ca\") pod \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") "
Oct 30 06:09:33.543076 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.542990    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-openshift-ca\") pod \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") "
Oct 30 06:09:33.543160 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.543081    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-kube-api-access-hmh9j\" (UniqueName: \"kubernetes.io/projected/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-kube-api-access-hmh9j\") pod \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\" (UID: \"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a\") "
Oct 30 06:09:33.543987 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.543946    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-openshift-ca" pod "9a02a5a8-acf1-47a1-a43e-21d9f90cca9a" (UID: "9a02a5a8-acf1-47a1-a43e-21d9f90cca9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:33.546277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.546250    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-xks-ca" pod "9a02a5a8-acf1-47a1-a43e-21d9f90cca9a" (UID: "9a02a5a8-acf1-47a1-a43e-21d9f90cca9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:09:33.546401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.546353    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-kube-api-access-hmh9j" pod "9a02a5a8-acf1-47a1-a43e-21d9f90cca9a" (UID: "9a02a5a8-acf1-47a1-a43e-21d9f90cca9a"). InnerVolumeSpecName "kube-api-access-hmh9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:33.644689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.644659    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:33.644689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.644688    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmh9j\" (UniqueName: \"kubernetes.io/projected/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-kube-api-access-hmh9j\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:33.644868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.644699    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:33.713837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.713811    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"]
Oct 30 06:09:33.715636 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:09:33.715600    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fd6b8bf_29a5_49ca_906e_c4b7c4a20c12.slice/crio-da117082140bf36718f4cbb83fbcb4779a60f3888361ce49fc91688770d41023 WatchSource:0}: Error finding container da117082140bf36718f4cbb83fbcb4779a60f3888361ce49fc91688770d41023: Status 404 returned error can't find the container with id da117082140bf36718f4cbb83fbcb4779a60f3888361ce49fc91688770d41023
Oct 30 06:09:33.995154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:33.995047    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:34.218391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.218363    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"
Oct 30 06:09:34.218788 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.218361    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn" event={"ID":"9a02a5a8-acf1-47a1-a43e-21d9f90cca9a","Type":"ContainerDied","Data":"6e678bd8714ebeeb60d567eeb2b3d9aaa2ab8879afb9fac959ae5e7b12e8c318"}
Oct 30 06:09:34.218788 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.218486    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e678bd8714ebeeb60d567eeb2b3d9aaa2ab8879afb9fac959ae5e7b12e8c318"
Oct 30 06:09:34.220358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.220332    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6" event={"ID":"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12","Type":"ContainerStarted","Data":"b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4"}
Oct 30 06:09:34.220515 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.220372    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6" event={"ID":"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12","Type":"ContainerStarted","Data":"da117082140bf36718f4cbb83fbcb4779a60f3888361ce49fc91688770d41023"}
Oct 30 06:09:34.259515 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.259436    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6" podStartSLOduration=1.259422377 podStartE2EDuration="1.259422377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:34.256990648 +0000 UTC m=+2289.088027862" watchObservedRunningTime="2026-10-30 06:09:34.259422377 +0000 UTC m=+2289.090459593"
Oct 30 06:09:34.379543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.379513    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:34.606846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.606742    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl" podUID="5ce90f27-00c2-46df-b6f9-87a760f71466" containerName="payload-processing" containerID="cri-o://b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c" gracePeriod=25
Oct 30 06:09:34.640341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.640302    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9" podUID="ba1f98e0-9947-4419-9a72-40efa3015dae" containerName="payload-pre-processing" containerID="cri-o://541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2" gracePeriod=25
Oct 30 06:09:34.888454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:34.888427    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:09:35.066383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.066355    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce90f27-00c2-46df-b6f9-87a760f71466-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce90f27-00c2-46df-b6f9-87a760f71466-plugins-config-volume\") pod \"5ce90f27-00c2-46df-b6f9-87a760f71466\" (UID: \"5ce90f27-00c2-46df-b6f9-87a760f71466\") "
Oct 30 06:09:35.066567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.066511    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce90f27-00c2-46df-b6f9-87a760f71466-kube-api-access-8hhq4\" (UniqueName: \"kubernetes.io/projected/5ce90f27-00c2-46df-b6f9-87a760f71466-kube-api-access-8hhq4\") pod \"5ce90f27-00c2-46df-b6f9-87a760f71466\" (UID: \"5ce90f27-00c2-46df-b6f9-87a760f71466\") "
Oct 30 06:09:35.066922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.066897    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce90f27-00c2-46df-b6f9-87a760f71466-plugins-config-volume" pod "5ce90f27-00c2-46df-b6f9-87a760f71466" (UID: "5ce90f27-00c2-46df-b6f9-87a760f71466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:35.067209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.067162    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce90f27-00c2-46df-b6f9-87a760f71466-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:35.068896 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.068871    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce90f27-00c2-46df-b6f9-87a760f71466-kube-api-access-8hhq4" pod "5ce90f27-00c2-46df-b6f9-87a760f71466" (UID: "5ce90f27-00c2-46df-b6f9-87a760f71466"). InnerVolumeSpecName "kube-api-access-8hhq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:35.168187 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.168154    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hhq4\" (UniqueName: \"kubernetes.io/projected/5ce90f27-00c2-46df-b6f9-87a760f71466-kube-api-access-8hhq4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:35.228913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.228882    5250 generic.go:356] "Generic (PLEG): container finished" podID="5ce90f27-00c2-46df-b6f9-87a760f71466" containerID="b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c" exitCode=0
Oct 30 06:09:35.229332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.228941    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"
Oct 30 06:09:35.229332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.228968    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl" event={"ID":"5ce90f27-00c2-46df-b6f9-87a760f71466","Type":"ContainerDied","Data":"b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c"}
Oct 30 06:09:35.229332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.229044    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl" event={"ID":"5ce90f27-00c2-46df-b6f9-87a760f71466","Type":"ContainerDied","Data":"4ea76657b34b14ad2c73f5193af87e0e409572d1ca9bdeec5b12f0748909cca7"}
Oct 30 06:09:35.229332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.229067    5250 scope.go:122] "RemoveContainer" containerID="b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c"
Oct 30 06:09:35.241330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.241312    5250 scope.go:122] "RemoveContainer" containerID="b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c"
Oct 30 06:09:35.241603 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:35.241579    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c\": container with ID starting with b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c not found: ID does not exist" containerID="b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c"
Oct 30 06:09:35.241664 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.241614    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c"} err="failed to get container status \"b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c\": rpc error: code = NotFound desc = could not find container \"b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c\": container with ID starting with b235098ca88052fd036d446f02949b68f750d20b711d85c51dcc0303833e097c not found: ID does not exist"
Oct 30 06:09:35.281864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.281843    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"]
Oct 30 06:09:35.286717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.286697    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d917a1-866d988ff6-8xhjl"]
Oct 30 06:09:35.537743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.537649    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:35.950634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.950558    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:09:35.978642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978612    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-kube-api-access-c2hzm\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-kube-api-access-c2hzm\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.978822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978670    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-credential-socket\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.978822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978694    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b783f198-7706-41b2-8cec-b91b27576482-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783f198-7706-41b2-8cec-b91b27576482-istio-podinfo\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.978822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978724    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-socket\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.978822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978754    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-data\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.978822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978789    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-envoy\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.979167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978825    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-certs\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.979167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978909    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-istio-token\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-istio-token\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.979167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978990    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b783f198-7706-41b2-8cec-b91b27576482-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783f198-7706-41b2-8cec-b91b27576482-istiod-ca-cert\") pod \"b783f198-7706-41b2-8cec-b91b27576482\" (UID: \"b783f198-7706-41b2-8cec-b91b27576482\") "
Oct 30 06:09:35.979167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.978973    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-credential-socket" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:35.979167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.979053    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-socket" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:35.979167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.979136    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-data" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:35.979535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.979335    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-certs" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:35.979601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.979580    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b783f198-7706-41b2-8cec-b91b27576482-istiod-ca-cert" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:35.979817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.979605    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:35.979817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.979626    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:35.979817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.979637    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:35.979817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.979645    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:35.981450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.981427    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-istio-token" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:35.981450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.981428    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b783f198-7706-41b2-8cec-b91b27576482-istio-podinfo" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:09:35.981737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.981710    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-envoy" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:09:35.981737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:35.981722    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-kube-api-access-c2hzm" pod "b783f198-7706-41b2-8cec-b91b27576482" (UID: "b783f198-7706-41b2-8cec-b91b27576482"). InnerVolumeSpecName "kube-api-access-c2hzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:36.002420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.002395    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:36.080447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.080372    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2hzm\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-kube-api-access-c2hzm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:36.080447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.080396    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783f198-7706-41b2-8cec-b91b27576482-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:36.080447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.080407    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783f198-7706-41b2-8cec-b91b27576482-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:36.080447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.080415    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b783f198-7706-41b2-8cec-b91b27576482-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:36.080447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.080422    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783f198-7706-41b2-8cec-b91b27576482-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:36.238855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.238826    5250 generic.go:356] "Generic (PLEG): container finished" podID="b783f198-7706-41b2-8cec-b91b27576482" containerID="74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02" exitCode=0
Oct 30 06:09:36.239316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.238948    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"
Oct 30 06:09:36.239316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.238974    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" event={"ID":"b783f198-7706-41b2-8cec-b91b27576482","Type":"ContainerDied","Data":"74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02"}
Oct 30 06:09:36.239316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.239034    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk" event={"ID":"b783f198-7706-41b2-8cec-b91b27576482","Type":"ContainerDied","Data":"e4acd4e127bf035bde7f194ccb0d4697eee752687f4c877ee9b22ebecc596589"}
Oct 30 06:09:36.239316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.239066    5250 scope.go:122] "RemoveContainer" containerID="74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02"
Oct 30 06:09:36.251679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.251653    5250 scope.go:122] "RemoveContainer" containerID="d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34"
Oct 30 06:09:36.263346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.263325    5250 scope.go:122] "RemoveContainer" containerID="74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02"
Oct 30 06:09:36.263652 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:36.263632    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02\": container with ID starting with 74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02 not found: ID does not exist" containerID="74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02"
Oct 30 06:09:36.263696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.263662    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02"} err="failed to get container status \"74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02\": rpc error: code = NotFound desc = could not find container \"74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02\": container with ID starting with 74e009928a8b1bce7cf7e34d9789c15540a6330ebcdeff7587ce395ab559af02 not found: ID does not exist"
Oct 30 06:09:36.263696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.263682    5250 scope.go:122] "RemoveContainer" containerID="d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34"
Oct 30 06:09:36.263926 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:36.263907    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34\": container with ID starting with d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34 not found: ID does not exist" containerID="d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34"
Oct 30 06:09:36.263973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.263933    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34"} err="failed to get container status \"d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34\": rpc error: code = NotFound desc = could not find container \"d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34\": container with ID starting with d3a048649777084f30583df3d12b7be4edad667f3d1a76d7cc7df466c5876e34 not found: ID does not exist"
Oct 30 06:09:36.287273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.287139    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"]
Oct 30 06:09:36.297127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:36.297105    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-dm2dk"]
Oct 30 06:09:37.281960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:37.281923    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce90f27-00c2-46df-b6f9-87a760f71466" path="/var/lib/kubelet/pods/5ce90f27-00c2-46df-b6f9-87a760f71466/volumes"
Oct 30 06:09:37.282660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:37.282638    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b783f198-7706-41b2-8cec-b91b27576482" path="/var/lib/kubelet/pods/b783f198-7706-41b2-8cec-b91b27576482/volumes"
Oct 30 06:09:37.534379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:37.534297    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:39.039187 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:39.039137    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:40.218491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:40.218454    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"]
Oct 30 06:09:40.247314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:40.247281    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"]
Oct 30 06:09:40.382454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:40.382413    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:40.525085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:40.525006    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:40.884084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:40.884051    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:41.016062 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:41.016028    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9" podUID="ba1f98e0-9947-4419-9a72-40efa3015dae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:41.030346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:41.030317    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"]
Oct 30 06:09:41.040023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:41.039992    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77c3e5b-cht6f"]
Oct 30 06:09:41.282156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:41.282079    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8e890c-5572-4462-90a0-06733f199fbb" path="/var/lib/kubelet/pods/3f8e890c-5572-4462-90a0-06733f199fbb/volumes"
Oct 30 06:09:42.485093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:42.485056    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:43.378973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:43.378941    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:43.391728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:43.391703    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:43.951340 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:43.951307    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:44.101449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:44.101413    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:44.311559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:44.311479    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:09:44.315715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:44.315687    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:45.218884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.218850    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr" podUID="0bfdf965-fc79-496e-b6b6-a0c6e1807df8" containerName="payload-processing" containerID="cri-o://a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107" gracePeriod=25
Oct 30 06:09:45.248145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.248116    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk" podUID="6afab366-9076-4bd4-8fd5-220ca11b1f80" containerName="payload-pre-processing" containerID="cri-o://97c0fb0b8d58de3af39842554752b74665970a17d4eb0f1a9159b201973b82a4" gracePeriod=25
Oct 30 06:09:45.495023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.494999    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:09:45.600114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.600075    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-kube-api-access-sxmwr\" (UniqueName: \"kubernetes.io/projected/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-kube-api-access-sxmwr\") pod \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\" (UID: \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\") "
Oct 30 06:09:45.600332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.600315    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-plugins-config-volume\") pod \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\" (UID: \"0bfdf965-fc79-496e-b6b6-a0c6e1807df8\") "
Oct 30 06:09:45.600828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.600743    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-plugins-config-volume" pod "0bfdf965-fc79-496e-b6b6-a0c6e1807df8" (UID: "0bfdf965-fc79-496e-b6b6-a0c6e1807df8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:45.600828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.600858    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:45.602503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.602477    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-kube-api-access-sxmwr" pod "0bfdf965-fc79-496e-b6b6-a0c6e1807df8" (UID: "0bfdf965-fc79-496e-b6b6-a0c6e1807df8"). InnerVolumeSpecName "kube-api-access-sxmwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:45.702561 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.702528    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxmwr\" (UniqueName: \"kubernetes.io/projected/0bfdf965-fc79-496e-b6b6-a0c6e1807df8-kube-api-access-sxmwr\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:45.900076 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:45.900036    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:46.316894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.316813    5250 generic.go:356] "Generic (PLEG): container finished" podID="0bfdf965-fc79-496e-b6b6-a0c6e1807df8" containerID="a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107" exitCode=0
Oct 30 06:09:46.317339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.316902    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr" event={"ID":"0bfdf965-fc79-496e-b6b6-a0c6e1807df8","Type":"ContainerDied","Data":"a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107"}
Oct 30 06:09:46.317339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.316925    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"
Oct 30 06:09:46.317339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.316957    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr" event={"ID":"0bfdf965-fc79-496e-b6b6-a0c6e1807df8","Type":"ContainerDied","Data":"f059363d8f67e72415fc9abb7e8960d61eff2621170f0360253f0c63867bdad3"}
Oct 30 06:09:46.317339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.316981    5250 scope.go:122] "RemoveContainer" containerID="a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107"
Oct 30 06:09:46.328171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.328154    5250 scope.go:122] "RemoveContainer" containerID="a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107"
Oct 30 06:09:46.328512 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:46.328492    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107\": container with ID starting with a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107 not found: ID does not exist" containerID="a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107"
Oct 30 06:09:46.328572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.328520    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107"} err="failed to get container status \"a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107\": rpc error: code = NotFound desc = could not find container \"a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107\": container with ID starting with a586125a269230f18e33b8d40ba8fc7fb5c919f7bea6464d03fb286701184107 not found: ID does not exist"
Oct 30 06:09:46.365548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.365518    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"]
Oct 30 06:09:46.376027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:46.376002    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7dfcb7-555cd7cc5b-hfzkr"]
Oct 30 06:09:47.282117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:47.282088    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bfdf965-fc79-496e-b6b6-a0c6e1807df8" path="/var/lib/kubelet/pods/0bfdf965-fc79-496e-b6b6-a0c6e1807df8/volumes"
Oct 30 06:09:47.491197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:47.491150    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:49.092824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:49.092793    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:50.987974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:50.987913    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:51.015845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:51.015819    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9" podUID="ba1f98e0-9947-4419-9a72-40efa3015dae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:51.263992 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:51.263895    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk" podUID="6afab366-9076-4bd4-8fd5-220ca11b1f80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:09:52.774729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:52.774696    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:54.325135 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:54.325066    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:54.902766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:54.902708    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:09:54.949405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:54.949378    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1f98e0-9947-4419-9a72-40efa3015dae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1f98e0-9947-4419-9a72-40efa3015dae-plugins-config-volume\") pod \"ba1f98e0-9947-4419-9a72-40efa3015dae\" (UID: \"ba1f98e0-9947-4419-9a72-40efa3015dae\") "
Oct 30 06:09:54.949592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:54.949471    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1f98e0-9947-4419-9a72-40efa3015dae-kube-api-access-rv6lk\" (UniqueName: \"kubernetes.io/projected/ba1f98e0-9947-4419-9a72-40efa3015dae-kube-api-access-rv6lk\") pod \"ba1f98e0-9947-4419-9a72-40efa3015dae\" (UID: \"ba1f98e0-9947-4419-9a72-40efa3015dae\") "
Oct 30 06:09:54.949877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:54.949837    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1f98e0-9947-4419-9a72-40efa3015dae-plugins-config-volume" pod "ba1f98e0-9947-4419-9a72-40efa3015dae" (UID: "ba1f98e0-9947-4419-9a72-40efa3015dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:54.950244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:54.950219    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1f98e0-9947-4419-9a72-40efa3015dae-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:54.952444 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:54.952405    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1f98e0-9947-4419-9a72-40efa3015dae-kube-api-access-rv6lk" pod "ba1f98e0-9947-4419-9a72-40efa3015dae" (UID: "ba1f98e0-9947-4419-9a72-40efa3015dae"). InnerVolumeSpecName "kube-api-access-rv6lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:55.051032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.050994    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv6lk\" (UniqueName: \"kubernetes.io/projected/ba1f98e0-9947-4419-9a72-40efa3015dae-kube-api-access-rv6lk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:55.394116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.394076    5250 generic.go:356] "Generic (PLEG): container finished" podID="ba1f98e0-9947-4419-9a72-40efa3015dae" containerID="541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2" exitCode=0
Oct 30 06:09:55.394598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.394138    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"
Oct 30 06:09:55.394598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.394162    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9" event={"ID":"ba1f98e0-9947-4419-9a72-40efa3015dae","Type":"ContainerDied","Data":"541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2"}
Oct 30 06:09:55.394598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.394233    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9" event={"ID":"ba1f98e0-9947-4419-9a72-40efa3015dae","Type":"ContainerDied","Data":"81a78ad28a873bc687fc0355500a762ab5bbe7898f1025930509eb0d57950b7e"}
Oct 30 06:09:55.394598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.394260    5250 scope.go:122] "RemoveContainer" containerID="541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2"
Oct 30 06:09:55.404758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.404735    5250 scope.go:122] "RemoveContainer" containerID="541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2"
Oct 30 06:09:55.405040 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:55.405019    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2\": container with ID starting with 541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2 not found: ID does not exist" containerID="541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2"
Oct 30 06:09:55.405118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.405047    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2"} err="failed to get container status \"541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2\": rpc error: code = NotFound desc = could not find container \"541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2\": container with ID starting with 541587ccc8639f145eca8ee7d090d61e8b6a7dcc1d9c7ff714b7d762dfa2f5f2 not found: ID does not exist"
Oct 30 06:09:55.443474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.443444    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"]
Oct 30 06:09:55.451837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.451805    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d917a1-cdc48b67f-fwkz9"]
Oct 30 06:09:55.591257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.591006    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"]
Oct 30 06:09:55.591435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.591333    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45" podUID="e43f3c3a-3d30-4744-aaed-8d39c60f71e6" containerName="maas-api" containerID="cri-o://1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6" gracePeriod=30
Oct 30 06:09:55.681630 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.681589    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:55.689771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.689722    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45" podUID="e43f3c3a-3d30-4744-aaed-8d39c60f71e6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.163:8443/health\": dial tcp 10.133.0.163:8443: connect: connection refused"
Oct 30 06:09:55.940635 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:55.940610    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:09:56.092218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.092139    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-openshift-service-ca\") pod \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") "
Oct 30 06:09:56.092440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.092233    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-maas-api-tls\") pod \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") "
Oct 30 06:09:56.092440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.092306    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-kube-api-access-j5pzf\" (UniqueName: \"kubernetes.io/projected/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-kube-api-access-j5pzf\") pod \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") "
Oct 30 06:09:56.092578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.092445    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-metrics-certs\") pod \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\" (UID: \"e43f3c3a-3d30-4744-aaed-8d39c60f71e6\") "
Oct 30 06:09:56.094528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.094466    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-openshift-service-ca" pod "e43f3c3a-3d30-4744-aaed-8d39c60f71e6" (UID: "e43f3c3a-3d30-4744-aaed-8d39c60f71e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:09:56.099118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.099070    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-kube-api-access-j5pzf" pod "e43f3c3a-3d30-4744-aaed-8d39c60f71e6" (UID: "e43f3c3a-3d30-4744-aaed-8d39c60f71e6"). InnerVolumeSpecName "kube-api-access-j5pzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:09:56.099379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.099357    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-metrics-certs" pod "e43f3c3a-3d30-4744-aaed-8d39c60f71e6" (UID: "e43f3c3a-3d30-4744-aaed-8d39c60f71e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:09:56.105474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.105444    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-maas-api-tls" pod "e43f3c3a-3d30-4744-aaed-8d39c60f71e6" (UID: "e43f3c3a-3d30-4744-aaed-8d39c60f71e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:09:56.194134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.194054    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pzf\" (UniqueName: \"kubernetes.io/projected/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-kube-api-access-j5pzf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:56.194134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.194082    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:56.194134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.194095    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:56.194134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.194103    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43f3c3a-3d30-4744-aaed-8d39c60f71e6-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:09:56.273779 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.273746    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:56.404527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.404495    5250 generic.go:356] "Generic (PLEG): container finished" podID="e43f3c3a-3d30-4744-aaed-8d39c60f71e6" containerID="1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6" exitCode=0
Oct 30 06:09:56.404981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.404536    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"
Oct 30 06:09:56.404981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.404559    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45" event={"ID":"e43f3c3a-3d30-4744-aaed-8d39c60f71e6","Type":"ContainerDied","Data":"1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6"}
Oct 30 06:09:56.404981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.404607    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45" event={"ID":"e43f3c3a-3d30-4744-aaed-8d39c60f71e6","Type":"ContainerDied","Data":"b5caeabc5bce41aa1f1b1b736ab56de5dda27a5236500aeecc7119a6c84b1ea4"}
Oct 30 06:09:56.404981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.404629    5250 scope.go:122] "RemoveContainer" containerID="1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6"
Oct 30 06:09:56.416225 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.416210    5250 scope.go:122] "RemoveContainer" containerID="1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6"
Oct 30 06:09:56.416513 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:09:56.416494    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6\": container with ID starting with 1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6 not found: ID does not exist" containerID="1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6"
Oct 30 06:09:56.416566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.416527    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6"} err="failed to get container status \"1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6\": rpc error: code = NotFound desc = could not find container \"1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6\": container with ID starting with 1538d44961193678ef5fd427ead9c4d22de8c2cc9fcc1255caae46a8943a06a6 not found: ID does not exist"
Oct 30 06:09:56.425589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.425568    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:56.452892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.452826    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"]
Oct 30 06:09:56.456764 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.456742    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d917a1-8d7f996f9-4cr45"]
Oct 30 06:09:56.571316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.571266    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:56.981667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:56.981632    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:57.282646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:57.282567    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1f98e0-9947-4419-9a72-40efa3015dae" path="/var/lib/kubelet/pods/ba1f98e0-9947-4419-9a72-40efa3015dae/volumes"
Oct 30 06:09:57.283034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:57.283020    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43f3c3a-3d30-4744-aaed-8d39c60f71e6" path="/var/lib/kubelet/pods/e43f3c3a-3d30-4744-aaed-8d39c60f71e6/volumes"
Oct 30 06:09:58.066054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:58.066020    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:09:59.527694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:09:59.527638    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:00.262492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:00.262458    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:00.391655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:00.391621    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"]
Oct 30 06:10:00.401888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:00.401863    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7dfcb7-z79hn"]
Oct 30 06:10:00.405913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:00.405865    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:00.553815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:00.553730    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:00.950991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:00.950958    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:01.263092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:01.263007    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk" podUID="6afab366-9076-4bd4-8fd5-220ca11b1f80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:10:01.281896 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:01.281863    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a02a5a8-acf1-47a1-a43e-21d9f90cca9a" path="/var/lib/kubelet/pods/9a02a5a8-acf1-47a1-a43e-21d9f90cca9a/volumes"
Oct 30 06:10:02.301452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:02.301390    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:02.428990 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:10:02.428943    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:10:03.087864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:03.087830    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:03.297208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:03.297157    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:03.443803 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:03.443768    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:03.668828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:03.668794    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:03.999732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:03.999698    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:04.503771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.503743    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:04.885649 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.885607    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"]
Oct 30 06:10:04.886798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886767    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9a02a5a8-acf1-47a1-a43e-21d9f90cca9a" containerName="revoke-keys"
Oct 30 06:10:04.886798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886798    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a02a5a8-acf1-47a1-a43e-21d9f90cca9a" containerName="revoke-keys"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886829    5250 cpu_manager.go:414] "containerMap: removing container" podUID="5ce90f27-00c2-46df-b6f9-87a760f71466" containerName="payload-processing"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886837    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce90f27-00c2-46df-b6f9-87a760f71466" containerName="payload-processing"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886848    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886855    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886864    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e43f3c3a-3d30-4744-aaed-8d39c60f71e6" containerName="maas-api"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886870    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43f3c3a-3d30-4744-aaed-8d39c60f71e6" containerName="maas-api"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886892    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0bfdf965-fc79-496e-b6b6-a0c6e1807df8" containerName="payload-processing"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886899    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bfdf965-fc79-496e-b6b6-a0c6e1807df8" containerName="payload-processing"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886916    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886923    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886937    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ba1f98e0-9947-4419-9a72-40efa3015dae" containerName="payload-pre-processing"
Oct 30 06:10:04.887021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.886945    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1f98e0-9947-4419-9a72-40efa3015dae" containerName="payload-pre-processing"
Oct 30 06:10:04.887741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.887094    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a02a5a8-acf1-47a1-a43e-21d9f90cca9a" containerName="revoke-keys"
Oct 30 06:10:04.887741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.887112    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce90f27-00c2-46df-b6f9-87a760f71466" containerName="payload-processing"
Oct 30 06:10:04.887741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.887122    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy"
Oct 30 06:10:04.887741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.887128    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43f3c3a-3d30-4744-aaed-8d39c60f71e6" containerName="maas-api"
Oct 30 06:10:04.887741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.887140    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bfdf965-fc79-496e-b6b6-a0c6e1807df8" containerName="payload-processing"
Oct 30 06:10:04.887741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.887148    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b783f198-7706-41b2-8cec-b91b27576482" containerName="istio-proxy"
Oct 30 06:10:04.887741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.887155    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1f98e0-9947-4419-9a72-40efa3015dae" containerName="payload-pre-processing"
Oct 30 06:10:04.893009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.892985    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:04.916710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.916687    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-720e9c-openshift-default-dockercfg-ldszj\""
Oct 30 06:10:04.923227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.922781    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"]
Oct 30 06:10:04.966584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:04.966528    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:05.012279 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012237    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-socket\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.012279 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012284    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c353321f-4449-4038-9811-5edea86b7ed5-istio-podinfo\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.012513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012427    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-credential-socket\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.012513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012457    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-istio-token\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.012513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012477    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-envoy\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.012513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012500    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbm7\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-kube-api-access-9pbm7\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.012721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012607    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-data\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.012721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012629    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-certs\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.012833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.012784    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c353321f-4449-4038-9811-5edea86b7ed5-istiod-ca-cert\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.116376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.115981    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-certs\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.116376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.116090    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c353321f-4449-4038-9811-5edea86b7ed5-istiod-ca-cert\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.116376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.116248    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-socket\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.116376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.116291    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c353321f-4449-4038-9811-5edea86b7ed5-istio-podinfo\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.116376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.116407    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-credential-socket\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.116376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.116445    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-istio-token\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.116376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.116471    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-envoy\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.117013 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.116508    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbm7\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-kube-api-access-9pbm7\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.117013 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.116549    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-data\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.117147 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.117042    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-data\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.120284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.117326    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-certs\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.120284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.117964    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c353321f-4449-4038-9811-5edea86b7ed5-istiod-ca-cert\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.120284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.118231    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-socket\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.120284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.118983    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:05.120284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.119689    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-credential-socket\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.127831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.127803    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c353321f-4449-4038-9811-5edea86b7ed5-istio-podinfo\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.153758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.153682    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-istio-token\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.159917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.159857    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbm7\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-kube-api-access-9pbm7\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.160457 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.160428    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-envoy\") pod \"e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") " pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.214039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.213938    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:05.486403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.486355    5250 generic.go:356] "Generic (PLEG): container finished" podID="6afab366-9076-4bd4-8fd5-220ca11b1f80" containerID="97c0fb0b8d58de3af39842554752b74665970a17d4eb0f1a9159b201973b82a4" exitCode=0
Oct 30 06:10:05.486633 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.486526    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk" event={"ID":"6afab366-9076-4bd4-8fd5-220ca11b1f80","Type":"ContainerDied","Data":"97c0fb0b8d58de3af39842554752b74665970a17d4eb0f1a9159b201973b82a4"}
Oct 30 06:10:05.515572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.515540    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:10:05.555024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.554994    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"]
Oct 30 06:10:05.560375 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:10:05.559752    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc353321f_4449_4038_9811_5edea86b7ed5.slice/crio-3ccf70e66777dc22aa55f7c0a1bb0b781b6af79df75a76393c4f23a91f40e281 WatchSource:0}: Error finding container 3ccf70e66777dc22aa55f7c0a1bb0b781b6af79df75a76393c4f23a91f40e281: Status 404 returned error can't find the container with id 3ccf70e66777dc22aa55f7c0a1bb0b781b6af79df75a76393c4f23a91f40e281
Oct 30 06:10:05.562794 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.562759    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:05.562954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.562830    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:05.562954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.562862    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:05.610655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.610627    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:05.630273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.630226    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afab366-9076-4bd4-8fd5-220ca11b1f80-kube-api-access-rpnnv\" (UniqueName: \"kubernetes.io/projected/6afab366-9076-4bd4-8fd5-220ca11b1f80-kube-api-access-rpnnv\") pod \"6afab366-9076-4bd4-8fd5-220ca11b1f80\" (UID: \"6afab366-9076-4bd4-8fd5-220ca11b1f80\") "
Oct 30 06:10:05.631674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.631476    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6afab366-9076-4bd4-8fd5-220ca11b1f80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afab366-9076-4bd4-8fd5-220ca11b1f80-plugins-config-volume\") pod \"6afab366-9076-4bd4-8fd5-220ca11b1f80\" (UID: \"6afab366-9076-4bd4-8fd5-220ca11b1f80\") "
Oct 30 06:10:05.633729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.633703    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afab366-9076-4bd4-8fd5-220ca11b1f80-kube-api-access-rpnnv" pod "6afab366-9076-4bd4-8fd5-220ca11b1f80" (UID: "6afab366-9076-4bd4-8fd5-220ca11b1f80"). InnerVolumeSpecName "kube-api-access-rpnnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:10:05.635351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.635328    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afab366-9076-4bd4-8fd5-220ca11b1f80-plugins-config-volume" pod "6afab366-9076-4bd4-8fd5-220ca11b1f80" (UID: "6afab366-9076-4bd4-8fd5-220ca11b1f80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:10:05.734009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.733915    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpnnv\" (UniqueName: \"kubernetes.io/projected/6afab366-9076-4bd4-8fd5-220ca11b1f80-kube-api-access-rpnnv\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:10:05.734009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:05.733952    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afab366-9076-4bd4-8fd5-220ca11b1f80-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:10:06.499677 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:06.499641    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk" event={"ID":"6afab366-9076-4bd4-8fd5-220ca11b1f80","Type":"ContainerDied","Data":"bd6edb820b6351030839e55a92d049bc2dafa6d7e3a622b9a909658e5f192383"}
Oct 30 06:10:06.499677 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:06.499670    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"
Oct 30 06:10:06.500030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:06.499720    5250 scope.go:122] "RemoveContainer" containerID="97c0fb0b8d58de3af39842554752b74665970a17d4eb0f1a9159b201973b82a4"
Oct 30 06:10:06.502038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:06.502005    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" event={"ID":"c353321f-4449-4038-9811-5edea86b7ed5","Type":"ContainerStarted","Data":"2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73"}
Oct 30 06:10:06.502231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:06.502058    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" event={"ID":"c353321f-4449-4038-9811-5edea86b7ed5","Type":"ContainerStarted","Data":"3ccf70e66777dc22aa55f7c0a1bb0b781b6af79df75a76393c4f23a91f40e281"}
Oct 30 06:10:06.555762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:06.555710    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" podStartSLOduration=2.555694553 podStartE2EDuration="2.555694553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:06.554172223 +0000 UTC m=+2321.385209440" watchObservedRunningTime="2026-10-30 06:10:06.555694553 +0000 UTC m=+2321.386731771"
Oct 30 06:10:06.590684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:06.590646    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"]
Oct 30 06:10:06.595042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:06.595016    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7dfcb7-df9745dc9-g5bqk"]
Oct 30 06:10:07.215096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:07.215064    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:07.221267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:07.221244    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:07.282877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:07.282844    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6afab366-9076-4bd4-8fd5-220ca11b1f80" path="/var/lib/kubelet/pods/6afab366-9076-4bd4-8fd5-220ca11b1f80/volumes"
Oct 30 06:10:07.415905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:07.415826    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:07.511199 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:07.511100    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:07.512004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:07.511980    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:10:07.718472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:07.718102    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:07.879342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:07.879255    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:08.069127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:08.069097    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:09.173534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:09.173507    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:09.335937 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:09.335899    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:09.481696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:09.481601    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:10.031085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.031052    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"]
Oct 30 06:10:10.040456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.040432    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf6f60e-c45cp"]
Oct 30 06:10:10.552465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.552431    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"]
Oct 30 06:10:10.553418 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.553394    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6afab366-9076-4bd4-8fd5-220ca11b1f80" containerName="payload-pre-processing"
Oct 30 06:10:10.553418 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.553418    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afab366-9076-4bd4-8fd5-220ca11b1f80" containerName="payload-pre-processing"
Oct 30 06:10:10.553610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.553597    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afab366-9076-4bd4-8fd5-220ca11b1f80" containerName="payload-pre-processing"
Oct 30 06:10:10.558809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.558786    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.576259 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.575966    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-37bbcc\"/\"default-dockercfg-5tgw4\""
Oct 30 06:10:10.587933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.587908    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-37bbcc\"/\"kube-root-ca.crt\""
Oct 30 06:10:10.588468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.588446    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-37bbcc\"/\"auth-test-model-37bbcc-kserve-self-signed-certs\""
Oct 30 06:10:10.588669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.588654    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-37bbcc\"/\"openshift-service-ca.crt\""
Oct 30 06:10:10.593274 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.593252    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"]
Oct 30 06:10:10.649563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.649538    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:10.713745 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.713703    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tls-certs\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.713909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.713873    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-dshm\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.713977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.713954    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-home\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.714149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.714128    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdc2h\" (UniqueName: \"kubernetes.io/projected/48b2e21e-c385-43e1-a24c-b31f9958dcfa-kube-api-access-vdc2h\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.714341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.714320    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tmp-dir\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.714509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.714361    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-model-cache\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.815943 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.815855    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tls-certs\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.816841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.816815    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-dshm\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.817416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.817397    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-home\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.818011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.817989    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdc2h\" (UniqueName: \"kubernetes.io/projected/48b2e21e-c385-43e1-a24c-b31f9958dcfa-kube-api-access-vdc2h\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.818606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.818586    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tmp-dir\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.819122 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.819105    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-model-cache\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.819612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.817840    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-home\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.819758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.819481    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-model-cache\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.819864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.819042    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tmp-dir\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.822755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.822731    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tls-certs\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.823094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.823057    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-dshm\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.840755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.840699    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdc2h\" (UniqueName: \"kubernetes.io/projected/48b2e21e-c385-43e1-a24c-b31f9958dcfa-kube-api-access-vdc2h\") pod \"auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") " pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:10.845637 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.845605    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:10.873670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:10.873640    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:11.016679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.016635    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"]
Oct 30 06:10:11.022367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.022348    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:11.061015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.058468    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-720e9c\""
Oct 30 06:10:11.061015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.058514    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"]
Oct 30 06:10:11.063451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.063426    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"]
Oct 30 06:10:11.063563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.063513    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:11.077490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.077455    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-720e9c-dockercfg-qkqkd\""
Oct 30 06:10:11.093567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.093539    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"]
Oct 30 06:10:11.098642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.098615    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:11.124492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.124411    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313f229-0830-4703-bc0a-0419dde5f501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g\" (UID: \"3313f229-0830-4703-bc0a-0419dde5f501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:11.124683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.124660    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qsq\" (UniqueName: \"kubernetes.io/projected/3313f229-0830-4703-bc0a-0419dde5f501-kube-api-access-r7qsq\") pod \"payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g\" (UID: \"3313f229-0830-4703-bc0a-0419dde5f501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:11.230271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.230007    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qsq\" (UniqueName: \"kubernetes.io/projected/3313f229-0830-4703-bc0a-0419dde5f501-kube-api-access-r7qsq\") pod \"payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g\" (UID: \"3313f229-0830-4703-bc0a-0419dde5f501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:11.230271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.230075    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh\" (UID: \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:11.230271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.230166    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313f229-0830-4703-bc0a-0419dde5f501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g\" (UID: \"3313f229-0830-4703-bc0a-0419dde5f501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:11.230649 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.230275    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zlnj\" (UniqueName: \"kubernetes.io/projected/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-kube-api-access-7zlnj\") pod \"payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh\" (UID: \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:11.232324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.231513    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313f229-0830-4703-bc0a-0419dde5f501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g\" (UID: \"3313f229-0830-4703-bc0a-0419dde5f501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:11.238678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.238588    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"]
Oct 30 06:10:11.266651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.266616    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:11.269315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.269277    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qsq\" (UniqueName: \"kubernetes.io/projected/3313f229-0830-4703-bc0a-0419dde5f501-kube-api-access-r7qsq\") pod \"payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g\" (UID: \"3313f229-0830-4703-bc0a-0419dde5f501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:11.288248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.288219    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442c0652-4a4e-4e28-9422-490463c5eed4" path="/var/lib/kubelet/pods/442c0652-4a4e-4e28-9422-490463c5eed4/volumes"
Oct 30 06:10:11.334252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.331931    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlnj\" (UniqueName: \"kubernetes.io/projected/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-kube-api-access-7zlnj\") pod \"payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh\" (UID: \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:11.334252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.332133    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh\" (UID: \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:11.334252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.333072    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh\" (UID: \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:11.339407 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.339383    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:11.355480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.355457    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlnj\" (UniqueName: \"kubernetes.io/projected/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-kube-api-access-7zlnj\") pod \"payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh\" (UID: \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:11.380806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.380774    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:11.553988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.553948    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn" event={"ID":"48b2e21e-c385-43e1-a24c-b31f9958dcfa","Type":"ContainerStarted","Data":"6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f"}
Oct 30 06:10:11.554435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.554000    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn" event={"ID":"48b2e21e-c385-43e1-a24c-b31f9958dcfa","Type":"ContainerStarted","Data":"52137af4d69f13d63e75ca2bb6190d4e34f2d6068549146998217a58b0b9c81e"}
Oct 30 06:10:11.596242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.596121    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn" podStartSLOduration=1.596106421 podStartE2EDuration="1.596106421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:11.59352853 +0000 UTC m=+2326.424565750" watchObservedRunningTime="2026-10-30 06:10:11.596106421 +0000 UTC m=+2326.427143637"
Oct 30 06:10:11.666083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.666060    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"]
Oct 30 06:10:11.669337 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:10:11.669310    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3313f229_0830_4703_bc0a_0419dde5f501.slice/crio-7fcd29b9fdcde50459da2653b25e9fd66ccf524dc3cd1cd32ea663349c514802 WatchSource:0}: Error finding container 7fcd29b9fdcde50459da2653b25e9fd66ccf524dc3cd1cd32ea663349c514802: Status 404 returned error can't find the container with id 7fcd29b9fdcde50459da2653b25e9fd66ccf524dc3cd1cd32ea663349c514802
Oct 30 06:10:11.732402 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:10:11.726809    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f40f894_6054_4b5d_93d3_b8a91eb6abc8.slice/crio-00307e4a8c61ef5ca03772ff9936331fb95f44ef51ec8b16f1669eec1e2cc653 WatchSource:0}: Error finding container 00307e4a8c61ef5ca03772ff9936331fb95f44ef51ec8b16f1669eec1e2cc653: Status 404 returned error can't find the container with id 00307e4a8c61ef5ca03772ff9936331fb95f44ef51ec8b16f1669eec1e2cc653
Oct 30 06:10:11.732402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.728209    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"]
Oct 30 06:10:11.874005 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:11.873927    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:12.563892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:12.563850    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g" event={"ID":"3313f229-0830-4703-bc0a-0419dde5f501","Type":"ContainerStarted","Data":"9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353"}
Oct 30 06:10:12.564360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:12.563956    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:12.564360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:12.563978    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g" event={"ID":"3313f229-0830-4703-bc0a-0419dde5f501","Type":"ContainerStarted","Data":"7fcd29b9fdcde50459da2653b25e9fd66ccf524dc3cd1cd32ea663349c514802"}
Oct 30 06:10:12.565684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:12.565649    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh" event={"ID":"0f40f894-6054-4b5d-93d3-b8a91eb6abc8","Type":"ContainerStarted","Data":"0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4"}
Oct 30 06:10:12.565805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:12.565702    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh" event={"ID":"0f40f894-6054-4b5d-93d3-b8a91eb6abc8","Type":"ContainerStarted","Data":"00307e4a8c61ef5ca03772ff9936331fb95f44ef51ec8b16f1669eec1e2cc653"}
Oct 30 06:10:12.603253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:12.603076    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g" podStartSLOduration=2.60306062 podStartE2EDuration="2.60306062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:12.600924178 +0000 UTC m=+2327.431961397" watchObservedRunningTime="2026-10-30 06:10:12.60306062 +0000 UTC m=+2327.434097836"
Oct 30 06:10:12.639999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:12.639946    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh" podStartSLOduration=1.6399313709999999 podStartE2EDuration="1.639931371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:12.637309064 +0000 UTC m=+2327.468346281" watchObservedRunningTime="2026-10-30 06:10:12.639931371 +0000 UTC m=+2327.470968598"
Oct 30 06:10:13.028466 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:13.028432    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:13.574248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:13.574219    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:14.660082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:14.660051    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:16.210843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:16.210802    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:17.979657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:17.979627    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:19.586445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:19.586412    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:20.874370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:20.874332    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:20.887164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:20.887137    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:21.307972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:21.307869    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:21.654447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:21.654416    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:10:22.614605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:22.614555    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:22.816512 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:22.816478    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:23.168754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:23.168718    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:23.577802 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:23.577711    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:10:23.669843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:23.667013    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:23.825106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:23.825066    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:24.586414 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:24.586383    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:10:24.753350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:24.753291    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:26.376082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:26.376051    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:27.570349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:27.570310    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:27.719052 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:27.719013    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:28.087268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.087240    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"]
Oct 30 06:10:28.093172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.093155    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.122612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.122577    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"]
Oct 30 06:10:28.128460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.128432    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6b8f87-openshift-default-dockercfg-pn4kl\""
Oct 30 06:10:28.185529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.185494    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.185694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.185535    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.185739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.185700    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.185823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.185808    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.185867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.185840    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.185903 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.185869    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.185938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.185909    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsq7n\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-kube-api-access-vsq7n\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.185998 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.185986    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.186038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.186016    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.237106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.237071    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:28.287572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287537    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.287742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287607    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.287742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287631    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.287742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287711    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.287959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287788    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.287959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287812    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.287959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287846    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.287959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287874    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsq7n\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-kube-api-access-vsq7n\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.287959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287911    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.288349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.287999    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.288349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.288065    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.288349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.288200    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.288485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.288350    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.288485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.288455    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.290570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.290546    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.290704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.290685    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.309659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.309586    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsq7n\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-kube-api-access-vsq7n\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.309659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.309640    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.409746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.409716    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:28.733674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.733649    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"]
Oct 30 06:10:28.735475 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:10:28.735444    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072333b6_e21e_4adf_ab3c_c53e5169a9ea.slice/crio-3f1a53b9a89ae45eafecbf15953cb7fb78df7676a2615a5faa0218bd476f2cc1 WatchSource:0}: Error finding container 3f1a53b9a89ae45eafecbf15953cb7fb78df7676a2615a5faa0218bd476f2cc1: Status 404 returned error can't find the container with id 3f1a53b9a89ae45eafecbf15953cb7fb78df7676a2615a5faa0218bd476f2cc1
Oct 30 06:10:28.738568 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.738535    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:28.738659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.738607    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:28.738659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:28.738636    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:29.181834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.181802    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:29.328401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.328363    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:29.485319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.485233    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"]
Oct 30 06:10:29.538576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.538546    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"]
Oct 30 06:10:29.538774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.538654    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:29.569206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.569142    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-2f1e4c\""
Oct 30 06:10:29.581271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.579445    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-2f1e4c-dockercfg-c7xkt\""
Oct 30 06:10:29.634526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.634493    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce112f1-37d7-4061-bc37-dcab95805834-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq\" (UID: \"9ce112f1-37d7-4061-bc37-dcab95805834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:29.634702 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.634655    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54wc\" (UniqueName: \"kubernetes.io/projected/9ce112f1-37d7-4061-bc37-dcab95805834-kube-api-access-s54wc\") pod \"payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq\" (UID: \"9ce112f1-37d7-4061-bc37-dcab95805834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:29.723166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.723126    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" event={"ID":"072333b6-e21e-4adf-ab3c-c53e5169a9ea","Type":"ContainerStarted","Data":"af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff"}
Oct 30 06:10:29.723391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.723220    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" event={"ID":"072333b6-e21e-4adf-ab3c-c53e5169a9ea","Type":"ContainerStarted","Data":"3f1a53b9a89ae45eafecbf15953cb7fb78df7676a2615a5faa0218bd476f2cc1"}
Oct 30 06:10:29.735840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.735759    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce112f1-37d7-4061-bc37-dcab95805834-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq\" (UID: \"9ce112f1-37d7-4061-bc37-dcab95805834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:29.736338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.736098    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54wc\" (UniqueName: \"kubernetes.io/projected/9ce112f1-37d7-4061-bc37-dcab95805834-kube-api-access-s54wc\") pod \"payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq\" (UID: \"9ce112f1-37d7-4061-bc37-dcab95805834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:29.736620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.736603    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce112f1-37d7-4061-bc37-dcab95805834-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq\" (UID: \"9ce112f1-37d7-4061-bc37-dcab95805834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:29.758674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.758641    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54wc\" (UniqueName: \"kubernetes.io/projected/9ce112f1-37d7-4061-bc37-dcab95805834-kube-api-access-s54wc\") pod \"payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq\" (UID: \"9ce112f1-37d7-4061-bc37-dcab95805834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:29.786241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.782245    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" podStartSLOduration=1.782220769 podStartE2EDuration="1.782220769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:29.777689025 +0000 UTC m=+2344.608726255" watchObservedRunningTime="2026-10-30 06:10:29.782220769 +0000 UTC m=+2344.613257986"
Oct 30 06:10:29.858425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:29.858390    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:30.118438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.118342    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:30.197712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.196555    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"]
Oct 30 06:10:30.201421 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:10:30.201357    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce112f1_37d7_4061_bc37_dcab95805834.slice/crio-b3a716cc888663efc60ddf9ca8534514b684648a6d80d56ba7235336470ffdbb WatchSource:0}: Error finding container b3a716cc888663efc60ddf9ca8534514b684648a6d80d56ba7235336470ffdbb: Status 404 returned error can't find the container with id b3a716cc888663efc60ddf9ca8534514b684648a6d80d56ba7235336470ffdbb
Oct 30 06:10:30.410072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.410044    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:30.411633 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.411609    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 06:10:30.411744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.411658    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 06:10:30.733031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.732931    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq" event={"ID":"9ce112f1-37d7-4061-bc37-dcab95805834","Type":"ContainerStarted","Data":"0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd"}
Oct 30 06:10:30.733031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.732988    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq" event={"ID":"9ce112f1-37d7-4061-bc37-dcab95805834","Type":"ContainerStarted","Data":"b3a716cc888663efc60ddf9ca8534514b684648a6d80d56ba7235336470ffdbb"}
Oct 30 06:10:30.733031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.733022    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:30.774205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:30.771633    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq" podStartSLOduration=1.771614356 podStartE2EDuration="1.771614356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:30.770255425 +0000 UTC m=+2345.601292645" watchObservedRunningTime="2026-10-30 06:10:30.771614356 +0000 UTC m=+2345.602651577"
Oct 30 06:10:31.410356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:31.410322    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 06:10:31.410554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:31.410375    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 06:10:31.782157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:31.777604    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:32.399512 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:10:32.399473    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:10:32.410275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:32.410247    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 06:10:32.410675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:32.410297    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 06:10:33.410670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:33.410631    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 06:10:33.411357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:33.411320    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 06:10:33.539824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:33.539791    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:34.412593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:34.412553    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 06:10:34.413244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:34.413213    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 06:10:35.410052 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:35.410009    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 06:10:35.410285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:35.410087    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 06:10:35.541665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:35.541624    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:35.713437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:35.713339    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:36.416635 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:36.416600    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:36.417032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:36.417009    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:36.417801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:36.417783    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:10:37.210204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:37.208009    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:39.087746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:39.087701    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:40.780984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:40.780943    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:41.744553 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:41.744518    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:10:42.573618 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:42.573585    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:42.712557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:42.712526    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:44.102910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:44.102874    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:44.306669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:44.306630    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:45.970800 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:45.970765    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:47.807840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:47.807805    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:48.635341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:48.635307    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:48.788934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:48.788899    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:49.518331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:49.518298    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:51.443513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:51.443481    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:51.697650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:51.697562    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"]
Oct 30 06:10:51.748242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:51.747389    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:53.585072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:53.585041    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:55.361856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.361822    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:55.771797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.771763    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"]
Oct 30 06:10:55.776212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.776160    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.789475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.789453    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-452a59-openshift-default-dockercfg-wgvvb\""
Oct 30 06:10:55.804837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.804812    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"]
Oct 30 06:10:55.897223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897160    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.897223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897226    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.897467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897328    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n6ts\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-kube-api-access-9n6ts\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.897467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897363    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.897467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897448    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.897589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897574    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.897633 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897600    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb46152-58fd-46bd-9d89-14ba2e28208e-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.897732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897712    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.897772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.897751    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999084    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999125    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999190    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n6ts\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-kube-api-access-9n6ts\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999224    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999274    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999428    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999461    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb46152-58fd-46bd-9d89-14ba2e28208e-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999522    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999553    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999579    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999713    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:55.999918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:55.999901    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:56.000069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.000042    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:56.000294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.000265    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb46152-58fd-46bd-9d89-14ba2e28208e-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:56.002095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.002068    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:56.002397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.002377    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:56.039509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.039449    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:56.039607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.039570    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n6ts\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-kube-api-access-9n6ts\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:56.091480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.091454    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:56.420537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.420510    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"]
Oct 30 06:10:56.421916 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:10:56.421891    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb46152_58fd_46bd_9d89_14ba2e28208e.slice/crio-b44786f30128e6f2a85c3ea1077c9437cb98222fe390565cfa3ff635cc68342d WatchSource:0}: Error finding container b44786f30128e6f2a85c3ea1077c9437cb98222fe390565cfa3ff635cc68342d: Status 404 returned error can't find the container with id b44786f30128e6f2a85c3ea1077c9437cb98222fe390565cfa3ff635cc68342d
Oct 30 06:10:56.424587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.424554    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:56.424675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.424620    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:56.424675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.424649    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:10:56.873919 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.873827    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:56.987290 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.987233    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" event={"ID":"bdb46152-58fd-46bd-9d89-14ba2e28208e","Type":"ContainerStarted","Data":"2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980"}
Oct 30 06:10:56.987290 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:56.987288    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" event={"ID":"bdb46152-58fd-46bd-9d89-14ba2e28208e","Type":"ContainerStarted","Data":"b44786f30128e6f2a85c3ea1077c9437cb98222fe390565cfa3ff635cc68342d"}
Oct 30 06:10:57.042659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:57.042613    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" podStartSLOduration=2.042600069 podStartE2EDuration="2.042600069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:57.039045228 +0000 UTC m=+2371.870082445" watchObservedRunningTime="2026-10-30 06:10:57.042600069 +0000 UTC m=+2371.873637285"
Oct 30 06:10:57.077203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:57.070771    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:57.100238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:57.091832    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:10:57.100238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:57.094793    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 06:10:57.100238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:57.094850    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 06:10:58.091857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:58.091826    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 06:10:58.092517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:58.091880    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 06:10:58.801353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:58.801323    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:10:59.092072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:59.091976    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 06:10:59.092072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:10:59.092035    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 06:11:00.092502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:00.092468    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 06:11:00.092938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:00.092523    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 06:11:00.390442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:00.390410    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:01.092159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:01.092122    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 06:11:01.092423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:01.092197    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 06:11:01.939741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:01.939710    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:02.090204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:02.089949    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:02.092066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:02.092037    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 06:11:02.092377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:02.092342    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 06:11:02.302120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:02.301970    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:02.389426 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:11:02.389378    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:11:03.130497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:03.130469    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:11:03.131029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:03.130973    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:11:03.133539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:03.133501    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:11:03.685768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:03.685734    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:05.185861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:05.185819    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:06.719505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:06.719466    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6" podUID="0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" containerName="main" containerID="cri-o://b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4" gracePeriod=45
Oct 30 06:11:06.824695 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:06.824643    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:07.003088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.003062    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:11:07.060359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.060328    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:07.072633 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.072605    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tmp-dir\") pod \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") "
Oct 30 06:11:07.072822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.072717    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-home\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-home\") pod \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") "
Oct 30 06:11:07.072822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.072757    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-kube-api-access-c8v6n\" (UniqueName: \"kubernetes.io/projected/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-kube-api-access-c8v6n\") pod \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") "
Oct 30 06:11:07.072822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.072777    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tls-certs\") pod \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") "
Oct 30 06:11:07.072822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.072809    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-dshm\") pod \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") "
Oct 30 06:11:07.073071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.072844    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-model-cache\") pod \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\" (UID: \"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12\") "
Oct 30 06:11:07.073071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.072934    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tmp-dir" pod "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" (UID: "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:07.074064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.073406    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:07.074064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.073610    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-model-cache" pod "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" (UID: "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:07.074064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.073807    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-home" pod "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" (UID: "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:07.076063 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.076039    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-dshm" pod "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" (UID: "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:07.082931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.082903    5250 generic.go:356] "Generic (PLEG): container finished" podID="0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" containerID="b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4" exitCode=0
Oct 30 06:11:07.083073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.082959    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"
Oct 30 06:11:07.083073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.082997    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6" event={"ID":"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12","Type":"ContainerDied","Data":"b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4"}
Oct 30 06:11:07.083073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.083045    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6" event={"ID":"0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12","Type":"ContainerDied","Data":"da117082140bf36718f4cbb83fbcb4779a60f3888361ce49fc91688770d41023"}
Oct 30 06:11:07.083073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.083070    5250 scope.go:122] "RemoveContainer" containerID="b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4"
Oct 30 06:11:07.086576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.086551    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tls-certs" pod "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" (UID: "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:11:07.088321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.088298    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-kube-api-access-c8v6n" pod "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" (UID: "0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12"). InnerVolumeSpecName "kube-api-access-c8v6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:11:07.112116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.112091    5250 scope.go:122] "RemoveContainer" containerID="b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4"
Oct 30 06:11:07.112466 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:11:07.112442    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4\": container with ID starting with b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4 not found: ID does not exist" containerID="b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4"
Oct 30 06:11:07.112549 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.112478    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4"} err="failed to get container status \"b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4\": rpc error: code = NotFound desc = could not find container \"b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4\": container with ID starting with b5ea6a84d0074b533280edcf1bd290c3c2db203de321674fd43b82aec23707a4 not found: ID does not exist"
Oct 30 06:11:07.175003 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.174971    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:07.175003 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.175003    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8v6n\" (UniqueName: \"kubernetes.io/projected/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-kube-api-access-c8v6n\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:07.175237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.175018    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:07.175237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.175030    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:07.175237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.175043    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:07.212901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.212864    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:07.428526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.428495    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"]
Oct 30 06:11:07.438737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:07.438711    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d96a1f/auth-test-model-d96a1f-kserve-b486749f7-sjwr6"]
Oct 30 06:11:08.592654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:08.591256    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:09.282791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:09.282756    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" path="/var/lib/kubelet/pods/0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12/volumes"
Oct 30 06:11:10.267074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:10.267033    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:11.852754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:11.852720    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:13.495569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:13.495532    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:15.058057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:15.058014    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:16.578692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:16.578639    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:16.938874 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:16.938839    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"]
Oct 30 06:11:17.013068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:17.013029    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:17.651883 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:17.651852    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:18.133043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:18.133016    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:18.309567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:18.309532    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:18.515946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:18.515854    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:20.368641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:20.368596    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:22.174829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:22.174794    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:22.449109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:22.449022    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:22.675670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:22.675634    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:23.323838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:23.323806    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"]
Oct 30 06:11:23.324253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:23.324006    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy" containerID="cri-o://9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42" gracePeriod=30
Oct 30 06:11:24.041757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.041725    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:24.182134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.182101    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"]
Oct 30 06:11:24.182741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.182727    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" containerName="main"
Oct 30 06:11:24.182796 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.182744    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" containerName="main"
Oct 30 06:11:24.182880 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.182871    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd6b8bf-29a5-49ca-906e-c4b7c4a20c12" containerName="main"
Oct 30 06:11:24.186540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.186511    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:24.201405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.201382    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c201cd\""
Oct 30 06:11:24.217703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.217669    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:24.217851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.217826    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c201cd-dockercfg-shmzn\""
Oct 30 06:11:24.224332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.224304    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"]
Oct 30 06:11:24.317708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.317630    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"]
Oct 30 06:11:24.321921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.321903    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:24.335560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.335535    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nm8\" (UniqueName: \"kubernetes.io/projected/febdd55a-454d-472c-8ba7-fa5d99b48ea0-kube-api-access-q5nm8\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n\" (UID: \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:24.335853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.335572    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/febdd55a-454d-472c-8ba7-fa5d99b48ea0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n\" (UID: \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:24.342500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.342478    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"]
Oct 30 06:11:24.372935 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.372901    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:24.436645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.436611    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nm8\" (UniqueName: \"kubernetes.io/projected/febdd55a-454d-472c-8ba7-fa5d99b48ea0-kube-api-access-q5nm8\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n\" (UID: \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:24.436805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.436664    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/febdd55a-454d-472c-8ba7-fa5d99b48ea0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n\" (UID: \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:24.436867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.436838    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xrm\" (UniqueName: \"kubernetes.io/projected/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-kube-api-access-g6xrm\") pod \"payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh\" (UID: \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:24.436958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.436940    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh\" (UID: \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:24.437559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.437538    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/febdd55a-454d-472c-8ba7-fa5d99b48ea0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n\" (UID: \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:24.458393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.458343    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nm8\" (UniqueName: \"kubernetes.io/projected/febdd55a-454d-472c-8ba7-fa5d99b48ea0-kube-api-access-q5nm8\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n\" (UID: \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:24.501059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.501028    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:24.538293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.538257    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh\" (UID: \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:24.539772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.539744    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xrm\" (UniqueName: \"kubernetes.io/projected/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-kube-api-access-g6xrm\") pod \"payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh\" (UID: \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:24.557206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.539411    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh\" (UID: \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:24.601261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.601127    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xrm\" (UniqueName: \"kubernetes.io/projected/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-kube-api-access-g6xrm\") pod \"payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh\" (UID: \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:24.661931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:24.636521    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:25.220734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:25.220707    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"]
Oct 30 06:11:25.453640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:25.453599    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n" event={"ID":"febdd55a-454d-472c-8ba7-fa5d99b48ea0","Type":"ContainerStarted","Data":"3a7817caba4143f6989985f668e9b3a5b048e5e68e7b54796fa3b0bd59dd60f2"}
Oct 30 06:11:25.464988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:25.464910    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"]
Oct 30 06:11:25.829053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:25.828982    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:11:25.861245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:25.861194    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:25.944620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:25.944595    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:11:26.320058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.320014    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:26.393771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.393719    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n" event={"ID":"febdd55a-454d-472c-8ba7-fa5d99b48ea0","Type":"ContainerStarted","Data":"d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd"}
Oct 30 06:11:26.393990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.393806    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:26.395581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.395555    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh" event={"ID":"1ccb90a7-36f7-47a9-aa7b-d2a280bba926","Type":"ContainerStarted","Data":"f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb"}
Oct 30 06:11:26.395730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.395601    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh" event={"ID":"1ccb90a7-36f7-47a9-aa7b-d2a280bba926","Type":"ContainerStarted","Data":"3d099accef913952d205ab8f19db3dab6f835f7ab55785ad46a288fc1a22d761"}
Oct 30 06:11:26.395783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.395733    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:26.435258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.435198    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n" podStartSLOduration=2.435165627 podStartE2EDuration="2.435165627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:26.430466185 +0000 UTC m=+2401.261503402" watchObservedRunningTime="2026-10-30 06:11:26.435165627 +0000 UTC m=+2401.266202873"
Oct 30 06:11:26.583263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.582674    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:26.886438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:26.886399    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:27.403893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:27.403863    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:28.106151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.106113    5250 scope.go:122] "RemoveContainer" containerID="2eac04a2693ee3230ae145f3db2bce7099de52927cfb110ace1b02d2b5687ef3"
Oct 30 06:11:28.124712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.124683    5250 scope.go:122] "RemoveContainer" containerID="438741761fc3d6ddda37c87cb75703b5a689ec0bd40d517114327f671b0de418"
Oct 30 06:11:28.141266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.141239    5250 scope.go:122] "RemoveContainer" containerID="b57a5f4b4af6f12885b10f2dda1ad21237021eb8f844c74f2bc8a08f6eb5e21f"
Oct 30 06:11:28.210830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.210770    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh" podStartSLOduration=4.210753839 podStartE2EDuration="4.210753839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:26.486458399 +0000 UTC m=+2401.317495617" watchObservedRunningTime="2026-10-30 06:11:28.210753839 +0000 UTC m=+2403.041791055"
Oct 30 06:11:28.214671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.214645    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8871-dc2c62"]
Oct 30 06:11:28.220694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.220666    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:28.237332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.237309    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8871-dc2c62"]
Oct 30 06:11:28.238945 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.238925    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bnq\""
Oct 30 06:11:28.320509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.320416    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb9rz\" (UniqueName: \"kubernetes.io/projected/536dc605-cb14-49a7-bca5-bd6779b2d0b6-kube-api-access-kb9rz\") pod \"test-curl-8871-dc2c62\" (UID: \"536dc605-cb14-49a7-bca5-bd6779b2d0b6\") " pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:28.422100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.421811    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb9rz\" (UniqueName: \"kubernetes.io/projected/536dc605-cb14-49a7-bca5-bd6779b2d0b6-kube-api-access-kb9rz\") pod \"test-curl-8871-dc2c62\" (UID: \"536dc605-cb14-49a7-bca5-bd6779b2d0b6\") " pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:28.444760 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.444723    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb9rz\" (UniqueName: \"kubernetes.io/projected/536dc605-cb14-49a7-bca5-bd6779b2d0b6-kube-api-access-kb9rz\") pod \"test-curl-8871-dc2c62\" (UID: \"536dc605-cb14-49a7-bca5-bd6779b2d0b6\") " pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:28.524927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.524836    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:28.540545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.540514    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:28.748980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.748952    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.845554    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-token\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-token\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.845633    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea411ab-37fe-4d3e-bada-24d67576140c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea411ab-37fe-4d3e-bada-24d67576140c-istiod-ca-cert\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.845696    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-certs\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.845771    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-data\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.845835    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-credential-socket\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.845956    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-podinfo\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.846048    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-envoy\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.846077    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-socket\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.847171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.846113    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-kube-api-access-r2ksq\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-kube-api-access-r2ksq\") pod \"6ea411ab-37fe-4d3e-bada-24d67576140c\" (UID: \"6ea411ab-37fe-4d3e-bada-24d67576140c\") "
Oct 30 06:11:28.848169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.848144    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-data" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:28.850490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.849759    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-certs" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:28.850490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.849903    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-credential-socket" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:28.850490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.849955    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea411ab-37fe-4d3e-bada-24d67576140c-istiod-ca-cert" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:11:28.850490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.849955    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-socket" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:28.850490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.850074    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-kube-api-access-r2ksq" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "kube-api-access-r2ksq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:11:28.851762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.851727    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-podinfo" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:11:28.852159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.852125    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-envoy" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:28.853291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.853271    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-token" pod "6ea411ab-37fe-4d3e-bada-24d67576140c" (UID: "6ea411ab-37fe-4d3e-bada-24d67576140c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947723    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947750    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947760    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947768    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2ksq\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-kube-api-access-r2ksq\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947778    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947797    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea411ab-37fe-4d3e-bada-24d67576140c-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947805    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947813    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:28.948220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:28.947821    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea411ab-37fe-4d3e-bada-24d67576140c-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:29.012835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.012807    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8871-dc2c62"]
Oct 30 06:11:29.014445 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:11:29.014419    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod536dc605_cb14_49a7_bca5_bd6779b2d0b6.slice/crio-6efbd94e268f487bc7c665971a4386c2409eebf20741f13542268c4d22f4432e WatchSource:0}: Error finding container 6efbd94e268f487bc7c665971a4386c2409eebf20741f13542268c4d22f4432e: Status 404 returned error can't find the container with id 6efbd94e268f487bc7c665971a4386c2409eebf20741f13542268c4d22f4432e
Oct 30 06:11:29.425439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.425403    5250 generic.go:356] "Generic (PLEG): container finished" podID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerID="9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42" exitCode=0
Oct 30 06:11:29.425870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.425523    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" event={"ID":"6ea411ab-37fe-4d3e-bada-24d67576140c","Type":"ContainerDied","Data":"9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42"}
Oct 30 06:11:29.425870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.425576    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj" event={"ID":"6ea411ab-37fe-4d3e-bada-24d67576140c","Type":"ContainerDied","Data":"7cd41e4ca2e2c4b4a8b5bc97c76bfccf16ac88962258fc832879e3770e6bd2d0"}
Oct 30 06:11:29.425870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.425606    5250 scope.go:122] "RemoveContainer" containerID="9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42"
Oct 30 06:11:29.425870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.425711    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"
Oct 30 06:11:29.428111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.428037    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8871-dc2c62" event={"ID":"536dc605-cb14-49a7-bca5-bd6779b2d0b6","Type":"ContainerStarted","Data":"b6d1d2009987c46eaa0b81c72f3c6cb075bfcf4bb315bd563b3e2a91115f7808"}
Oct 30 06:11:29.428111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.428076    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8871-dc2c62" event={"ID":"536dc605-cb14-49a7-bca5-bd6779b2d0b6","Type":"ContainerStarted","Data":"6efbd94e268f487bc7c665971a4386c2409eebf20741f13542268c4d22f4432e"}
Oct 30 06:11:29.442339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.441701    5250 scope.go:122] "RemoveContainer" containerID="90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f"
Oct 30 06:11:29.454430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.454404    5250 scope.go:122] "RemoveContainer" containerID="9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42"
Oct 30 06:11:29.454765 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:11:29.454733    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42\": container with ID starting with 9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42 not found: ID does not exist" containerID="9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42"
Oct 30 06:11:29.454853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.454774    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42"} err="failed to get container status \"9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42\": rpc error: code = NotFound desc = could not find container \"9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42\": container with ID starting with 9b836e313223c9c25347d4fef8e4f1076d79fbf5b4c192fd3367f2ce4e048c42 not found: ID does not exist"
Oct 30 06:11:29.454853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.454798    5250 scope.go:122] "RemoveContainer" containerID="90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f"
Oct 30 06:11:29.455079 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:11:29.455060    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f\": container with ID starting with 90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f not found: ID does not exist" containerID="90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f"
Oct 30 06:11:29.455139 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.455086    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f"} err="failed to get container status \"90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f\": rpc error: code = NotFound desc = could not find container \"90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f\": container with ID starting with 90d7e6da45f788298bf1a5e7ffdf8239227aa1d32f08d29fbe51e3691c11c78f not found: ID does not exist"
Oct 30 06:11:29.493464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.493428    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"]
Oct 30 06:11:29.502534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.502510    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-s4njj"]
Oct 30 06:11:29.516822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:29.516770    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8871-dc2c62" podStartSLOduration=1.516753612 podStartE2EDuration="1.516753612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:29.515520906 +0000 UTC m=+2404.346558117" watchObservedRunningTime="2026-10-30 06:11:29.516753612 +0000 UTC m=+2404.347790830"
Oct 30 06:11:30.371900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:30.370550    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:30.485202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:30.484242    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8871-dc2c62"]
Oct 30 06:11:30.522375 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:30.511612    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8871-dc2c62"]
Oct 30 06:11:31.288827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:31.288796    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" path="/var/lib/kubelet/pods/6ea411ab-37fe-4d3e-bada-24d67576140c/volumes"
Oct 30 06:11:31.414396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:31.414363    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:31.445820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:31.445775    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8871-dc2c62" podUID="536dc605-cb14-49a7-bca5-bd6779b2d0b6" containerName="test-curl-8871-dc2c62" containerID="cri-o://b6d1d2009987c46eaa0b81c72f3c6cb075bfcf4bb315bd563b3e2a91115f7808" gracePeriod=2
Oct 30 06:11:31.563744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:31.563648    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:31.972486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:31.972446    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn" podUID="48b2e21e-c385-43e1-a24c-b31f9958dcfa" containerName="main" containerID="cri-o://6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f" gracePeriod=45
Oct 30 06:11:32.269244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.269216    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:11:32.321957 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.321926    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tls-certs\") pod \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") "
Oct 30 06:11:32.322149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.321984    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b2e21e-c385-43e1-a24c-b31f9958dcfa-kube-api-access-vdc2h\" (UniqueName: \"kubernetes.io/projected/48b2e21e-c385-43e1-a24c-b31f9958dcfa-kube-api-access-vdc2h\") pod \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") "
Oct 30 06:11:32.322149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.322066    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-model-cache\") pod \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") "
Oct 30 06:11:32.322380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.322235    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-home\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-home\") pod \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") "
Oct 30 06:11:32.322380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.322364    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-dshm\") pod \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") "
Oct 30 06:11:32.322503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.322422    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tmp-dir\") pod \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\" (UID: \"48b2e21e-c385-43e1-a24c-b31f9958dcfa\") "
Oct 30 06:11:32.322563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.322545    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-model-cache" pod "48b2e21e-c385-43e1-a24c-b31f9958dcfa" (UID: "48b2e21e-c385-43e1-a24c-b31f9958dcfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:32.323546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.322794    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-home" pod "48b2e21e-c385-43e1-a24c-b31f9958dcfa" (UID: "48b2e21e-c385-43e1-a24c-b31f9958dcfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:32.323546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.323092    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:32.323546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.323117    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:32.323546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.323152    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tmp-dir" pod "48b2e21e-c385-43e1-a24c-b31f9958dcfa" (UID: "48b2e21e-c385-43e1-a24c-b31f9958dcfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:32.325803 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.325773    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tls-certs" pod "48b2e21e-c385-43e1-a24c-b31f9958dcfa" (UID: "48b2e21e-c385-43e1-a24c-b31f9958dcfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:11:32.325928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.325911    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b2e21e-c385-43e1-a24c-b31f9958dcfa-kube-api-access-vdc2h" pod "48b2e21e-c385-43e1-a24c-b31f9958dcfa" (UID: "48b2e21e-c385-43e1-a24c-b31f9958dcfa"). InnerVolumeSpecName "kube-api-access-vdc2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:11:32.326333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.326310    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-dshm" pod "48b2e21e-c385-43e1-a24c-b31f9958dcfa" (UID: "48b2e21e-c385-43e1-a24c-b31f9958dcfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:11:32.374585 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:11:32.374553    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:11:32.411456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.411405    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:32.425836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.425274    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:32.425836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.425442    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:32.425836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.425471    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b2e21e-c385-43e1-a24c-b31f9958dcfa-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:32.425836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.425486    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdc2h\" (UniqueName: \"kubernetes.io/projected/48b2e21e-c385-43e1-a24c-b31f9958dcfa-kube-api-access-vdc2h\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:32.461829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.461785    5250 generic.go:356] "Generic (PLEG): container finished" podID="48b2e21e-c385-43e1-a24c-b31f9958dcfa" containerID="6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f" exitCode=0
Oct 30 06:11:32.462031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.461852    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"
Oct 30 06:11:32.462031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.461870    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn" event={"ID":"48b2e21e-c385-43e1-a24c-b31f9958dcfa","Type":"ContainerDied","Data":"6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f"}
Oct 30 06:11:32.462031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.461919    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn" event={"ID":"48b2e21e-c385-43e1-a24c-b31f9958dcfa","Type":"ContainerDied","Data":"52137af4d69f13d63e75ca2bb6190d4e34f2d6068549146998217a58b0b9c81e"}
Oct 30 06:11:32.462031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.461941    5250 scope.go:122] "RemoveContainer" containerID="6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f"
Oct 30 06:11:32.480203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.479848    5250 scope.go:122] "RemoveContainer" containerID="6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f"
Oct 30 06:11:32.480312 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:11:32.480284    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f\": container with ID starting with 6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f not found: ID does not exist" containerID="6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f"
Oct 30 06:11:32.480382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.480325    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f"} err="failed to get container status \"6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f\": rpc error: code = NotFound desc = could not find container \"6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f\": container with ID starting with 6096ab008a977c13acb577bb95300e15c3444c618edba8fbfc91743c2118171f not found: ID does not exist"
Oct 30 06:11:32.515572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.515487    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"]
Oct 30 06:11:32.519219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:32.519161    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-37bbcc/auth-test-model-37bbcc-kserve-5d546cbb9c-vsjkn"]
Oct 30 06:11:33.042876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.042844    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:33.089231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.089171    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"]
Oct 30 06:11:33.090220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090191    5250 cpu_manager.go:414] "containerMap: removing container" podUID="48b2e21e-c385-43e1-a24c-b31f9958dcfa" containerName="main"
Oct 30 06:11:33.090220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090216    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b2e21e-c385-43e1-a24c-b31f9958dcfa" containerName="main"
Oct 30 06:11:33.090432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090248    5250 cpu_manager.go:414] "containerMap: removing container" podUID="536dc605-cb14-49a7-bca5-bd6779b2d0b6" containerName="test-curl-8871-dc2c62"
Oct 30 06:11:33.090432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090257    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536dc605-cb14-49a7-bca5-bd6779b2d0b6" containerName="test-curl-8871-dc2c62"
Oct 30 06:11:33.090432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090283    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy"
Oct 30 06:11:33.090432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090288    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy"
Oct 30 06:11:33.090432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090298    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy"
Oct 30 06:11:33.090432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090303    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy"
Oct 30 06:11:33.090432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090421    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy"
Oct 30 06:11:33.090432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090437    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b2e21e-c385-43e1-a24c-b31f9958dcfa" containerName="main"
Oct 30 06:11:33.090808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.090454    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="536dc605-cb14-49a7-bca5-bd6779b2d0b6" containerName="test-curl-8871-dc2c62"
Oct 30 06:11:33.095558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.095535    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"]
Oct 30 06:11:33.095686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.095639    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:33.096544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.096519    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea411ab-37fe-4d3e-bada-24d67576140c" containerName="istio-proxy"
Oct 30 06:11:33.100063 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.100044    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:33.113675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.113608    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-95498c\""
Oct 30 06:11:33.115329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.115267    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"]
Oct 30 06:11:33.121123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.121094    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"]
Oct 30 06:11:33.137020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.136984    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvg2h\" (UniqueName: \"kubernetes.io/projected/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-kube-api-access-mvg2h\") pod \"payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst\" (UID: \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:33.137249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.137054    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16498d8f-ea36-4a05-b012-13c320cfe29a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b\" (UID: \"16498d8f-ea36-4a05-b012-13c320cfe29a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:33.137312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.137275    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst\" (UID: \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:33.137405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.137385    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98c9k\" (UniqueName: \"kubernetes.io/projected/16498d8f-ea36-4a05-b012-13c320cfe29a-kube-api-access-98c9k\") pod \"payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b\" (UID: \"16498d8f-ea36-4a05-b012-13c320cfe29a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:33.194370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.194335    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:33.239557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.239510    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16498d8f-ea36-4a05-b012-13c320cfe29a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b\" (UID: \"16498d8f-ea36-4a05-b012-13c320cfe29a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:33.239768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.239697    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst\" (UID: \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:33.239845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.239803    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98c9k\" (UniqueName: \"kubernetes.io/projected/16498d8f-ea36-4a05-b012-13c320cfe29a-kube-api-access-98c9k\") pod \"payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b\" (UID: \"16498d8f-ea36-4a05-b012-13c320cfe29a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:33.239939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.239921    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvg2h\" (UniqueName: \"kubernetes.io/projected/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-kube-api-access-mvg2h\") pod \"payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst\" (UID: \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:33.240902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.240596    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16498d8f-ea36-4a05-b012-13c320cfe29a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b\" (UID: \"16498d8f-ea36-4a05-b012-13c320cfe29a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:33.241071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.241057    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst\" (UID: \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:33.276950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.275930    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98c9k\" (UniqueName: \"kubernetes.io/projected/16498d8f-ea36-4a05-b012-13c320cfe29a-kube-api-access-98c9k\") pod \"payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b\" (UID: \"16498d8f-ea36-4a05-b012-13c320cfe29a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:33.322252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.320952    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvg2h\" (UniqueName: \"kubernetes.io/projected/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-kube-api-access-mvg2h\") pod \"payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst\" (UID: \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:33.346194 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.346142    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b2e21e-c385-43e1-a24c-b31f9958dcfa" path="/var/lib/kubelet/pods/48b2e21e-c385-43e1-a24c-b31f9958dcfa/volumes"
Oct 30 06:11:33.422623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.422590    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:33.429112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.429086    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:33.474737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.474704    5250 generic.go:356] "Generic (PLEG): container finished" podID="536dc605-cb14-49a7-bca5-bd6779b2d0b6" containerID="b6d1d2009987c46eaa0b81c72f3c6cb075bfcf4bb315bd563b3e2a91115f7808" exitCode=137
Oct 30 06:11:33.731108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.731082    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:33.750519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.750436    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"]
Oct 30 06:11:33.756668 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.756642    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536dc605-cb14-49a7-bca5-bd6779b2d0b6-kube-api-access-kb9rz\" (UniqueName: \"kubernetes.io/projected/536dc605-cb14-49a7-bca5-bd6779b2d0b6-kube-api-access-kb9rz\") pod \"536dc605-cb14-49a7-bca5-bd6779b2d0b6\" (UID: \"536dc605-cb14-49a7-bca5-bd6779b2d0b6\") "
Oct 30 06:11:33.760351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.760323    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536dc605-cb14-49a7-bca5-bd6779b2d0b6-kube-api-access-kb9rz" pod "536dc605-cb14-49a7-bca5-bd6779b2d0b6" (UID: "536dc605-cb14-49a7-bca5-bd6779b2d0b6"). InnerVolumeSpecName "kube-api-access-kb9rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:11:33.765981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.765954    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"]
Oct 30 06:11:33.770499 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:11:33.770473    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ddf29f_f574_4cb0_8d21_06a4e12f6d4a.slice/crio-35c3edb0b8488ffe0be46ddb8f0046cc342e661d35f16e13330198e5b6670be1 WatchSource:0}: Error finding container 35c3edb0b8488ffe0be46ddb8f0046cc342e661d35f16e13330198e5b6670be1: Status 404 returned error can't find the container with id 35c3edb0b8488ffe0be46ddb8f0046cc342e661d35f16e13330198e5b6670be1
Oct 30 06:11:33.858881 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:33.858851    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb9rz\" (UniqueName: \"kubernetes.io/projected/536dc605-cb14-49a7-bca5-bd6779b2d0b6-kube-api-access-kb9rz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:34.479785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.479749    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:34.489798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.489765    5250 scope.go:122] "RemoveContainer" containerID="b6d1d2009987c46eaa0b81c72f3c6cb075bfcf4bb315bd563b3e2a91115f7808"
Oct 30 06:11:34.490043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.489868    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:34.493710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.493679    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst" event={"ID":"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a","Type":"ContainerStarted","Data":"4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889"}
Oct 30 06:11:34.493870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.493738    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst" event={"ID":"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a","Type":"ContainerStarted","Data":"35c3edb0b8488ffe0be46ddb8f0046cc342e661d35f16e13330198e5b6670be1"}
Oct 30 06:11:34.495399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.494926    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:34.497809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.497783    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b" event={"ID":"16498d8f-ea36-4a05-b012-13c320cfe29a","Type":"ContainerStarted","Data":"428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0"}
Oct 30 06:11:34.497938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.497832    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b" event={"ID":"16498d8f-ea36-4a05-b012-13c320cfe29a","Type":"ContainerStarted","Data":"7fae355992bf40945a0ce0874026a7da5a18c4a1bc2a55d6644057b22d5b68fd"}
Oct 30 06:11:34.498158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.498137    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:34.536800 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.536747    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst" podStartSLOduration=1.536732078 podStartE2EDuration="1.536732078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:34.532232678 +0000 UTC m=+2409.363269897" watchObservedRunningTime="2026-10-30 06:11:34.536732078 +0000 UTC m=+2409.367769301"
Oct 30 06:11:34.544099 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:11:34.544021    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8871-dc2c62\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="536dc605-cb14-49a7-bca5-bd6779b2d0b6" pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:34.584795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:34.584727    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b" podStartSLOduration=1.5847077600000001 podStartE2EDuration="1.58470776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:34.580846923 +0000 UTC m=+2409.411884153" watchObservedRunningTime="2026-10-30 06:11:34.58470776 +0000 UTC m=+2409.415744982"
Oct 30 06:11:35.282863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:35.282827    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536dc605-cb14-49a7-bca5-bd6779b2d0b6" path="/var/lib/kubelet/pods/536dc605-cb14-49a7-bca5-bd6779b2d0b6/volumes"
Oct 30 06:11:35.292753 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:11:35.292719    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8871-dc2c62\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="536dc605-cb14-49a7-bca5-bd6779b2d0b6" pod="opendatahub/test-curl-8871-dc2c62"
Oct 30 06:11:35.843748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:35.843637    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:36.446965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:36.446928    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:36.516810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:36.516779    5250 generic.go:356] "Generic (PLEG): container finished" podID="c353321f-4449-4038-9811-5edea86b7ed5" containerID="2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73" exitCode=0
Oct 30 06:11:36.517015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:36.516848    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" event={"ID":"c353321f-4449-4038-9811-5edea86b7ed5","Type":"ContainerDied","Data":"2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73"}
Oct 30 06:11:36.517368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:36.517350    5250 scope.go:122] "RemoveContainer" containerID="2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73"
Oct 30 06:11:36.519095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:36.519069    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:11:36.519158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:36.519139    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:11:36.519237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:36.519172    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:11:37.030781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:37.030750    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"]
Oct 30 06:11:37.034387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:37.034346    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab564fc-5srjh"]
Oct 30 06:11:37.283059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:37.282981    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599f341c-e18f-427c-9e76-22fce7270c85" path="/var/lib/kubelet/pods/599f341c-e18f-427c-9e76-22fce7270c85/volumes"
Oct 30 06:11:37.408267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:37.408237    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:11:37.408443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:37.408370    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:11:37.511910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:37.511875    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:11:37.527483 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:37.527449    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" event={"ID":"c353321f-4449-4038-9811-5edea86b7ed5","Type":"ContainerStarted","Data":"8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a"}
Oct 30 06:11:38.081236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:38.073636    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:38.214638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:38.214603    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:11:38.215845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:38.215817    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 06:11:38.215982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:38.215866    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 06:11:39.215194 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:39.215150    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 06:11:39.215626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:39.215239    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 06:11:39.807022 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:39.806983    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:40.215072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:40.215038    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 06:11:40.215275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:40.215097    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 06:11:41.215621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:41.215583    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 06:11:41.216238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:41.215645    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 06:11:41.714737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:41.714699    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:41.918696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:41.918661    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"]
Oct 30 06:11:41.923619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:41.923593    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:41.940324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:41.940267    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"]
Oct 30 06:11:41.946137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:41.946113    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:11:41.946304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:41.946285    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:11:42.001265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.001166    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdwr\" (UniqueName: \"kubernetes.io/projected/b66db160-1292-4851-b8cc-4e53182b0acd-kube-api-access-9rdwr\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.001408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.001340    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.001408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.001362    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.102491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.102456    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.102675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.102655    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdwr\" (UniqueName: \"kubernetes.io/projected/b66db160-1292-4851-b8cc-4e53182b0acd-kube-api-access-9rdwr\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.102844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.102828    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.103480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.103454    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.105106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.105083    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.124149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.124125    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdwr\" (UniqueName: \"kubernetes.io/projected/b66db160-1292-4851-b8cc-4e53182b0acd-kube-api-access-9rdwr\") pod \"maas-api-revoke-sub-79128713-7ntq5\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.214307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.214270    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 06:11:42.214519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.214334    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 06:11:42.242050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.242022    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:42.587304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:42.587268    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"]
Oct 30 06:11:43.215851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:43.215821    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 06:11:43.216072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:43.216050    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 06:11:43.520756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:43.520574    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:43.638211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:43.635768    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5" event={"ID":"b66db160-1292-4851-b8cc-4e53182b0acd","Type":"ContainerStarted","Data":"6287c720bc265b45af39f0bdf4683303173dd9d694c998179b94872e930d359c"}
Oct 30 06:11:43.638211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:43.635837    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5" event={"ID":"b66db160-1292-4851-b8cc-4e53182b0acd","Type":"ContainerStarted","Data":"ecf37780845cd46b751d0064c255c7d3fd98e55b1c624029d7b91f13e5d27486"}
Oct 30 06:11:43.674203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:43.671095    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:43.689341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:43.689160    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5" podStartSLOduration=2.6891181570000002 podStartE2EDuration="2.689118157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:43.683604105 +0000 UTC m=+2418.514641337" watchObservedRunningTime="2026-10-30 06:11:43.689118157 +0000 UTC m=+2418.520155375"
Oct 30 06:11:44.220002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:44.219972    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:11:44.220494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:44.220471    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:11:44.221423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:44.221387    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:11:44.417354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:44.417326    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:44.645467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:44.645436    5250 generic.go:356] "Generic (PLEG): container finished" podID="b66db160-1292-4851-b8cc-4e53182b0acd" containerID="6287c720bc265b45af39f0bdf4683303173dd9d694c998179b94872e930d359c" exitCode=0
Oct 30 06:11:44.645936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:44.645514    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5" event={"ID":"b66db160-1292-4851-b8cc-4e53182b0acd","Type":"ContainerDied","Data":"6287c720bc265b45af39f0bdf4683303173dd9d694c998179b94872e930d359c"}
Oct 30 06:11:45.212364 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.212326    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:45.511892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.511527    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:11:45.845524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.845495    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:45.868757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.868729    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-openshift-ca\") pod \"b66db160-1292-4851-b8cc-4e53182b0acd\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") "
Oct 30 06:11:45.868922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.868911    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66db160-1292-4851-b8cc-4e53182b0acd-kube-api-access-9rdwr\" (UniqueName: \"kubernetes.io/projected/b66db160-1292-4851-b8cc-4e53182b0acd-kube-api-access-9rdwr\") pod \"b66db160-1292-4851-b8cc-4e53182b0acd\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") "
Oct 30 06:11:45.868983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.868960    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-xks-ca\") pod \"b66db160-1292-4851-b8cc-4e53182b0acd\" (UID: \"b66db160-1292-4851-b8cc-4e53182b0acd\") "
Oct 30 06:11:45.869111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.869083    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-openshift-ca" pod "b66db160-1292-4851-b8cc-4e53182b0acd" (UID: "b66db160-1292-4851-b8cc-4e53182b0acd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:11:45.869439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.869420    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:45.871259 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.871238    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-xks-ca" pod "b66db160-1292-4851-b8cc-4e53182b0acd" (UID: "b66db160-1292-4851-b8cc-4e53182b0acd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:11:45.871341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.871269    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66db160-1292-4851-b8cc-4e53182b0acd-kube-api-access-9rdwr" pod "b66db160-1292-4851-b8cc-4e53182b0acd" (UID: "b66db160-1292-4851-b8cc-4e53182b0acd"). InnerVolumeSpecName "kube-api-access-9rdwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:11:45.970560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.970519    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdwr\" (UniqueName: \"kubernetes.io/projected/b66db160-1292-4851-b8cc-4e53182b0acd-kube-api-access-9rdwr\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:45.970560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:45.970552    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66db160-1292-4851-b8cc-4e53182b0acd-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:11:46.029763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:46.029724    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"]
Oct 30 06:11:46.033620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:46.033595    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd36098-nhmrb"]
Oct 30 06:11:46.520881 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:46.520855    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:11:46.662281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:46.662246    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"
Oct 30 06:11:46.662473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:46.662246    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5" event={"ID":"b66db160-1292-4851-b8cc-4e53182b0acd","Type":"ContainerDied","Data":"ecf37780845cd46b751d0064c255c7d3fd98e55b1c624029d7b91f13e5d27486"}
Oct 30 06:11:46.662473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:46.662391    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecf37780845cd46b751d0064c255c7d3fd98e55b1c624029d7b91f13e5d27486"
Oct 30 06:11:46.994388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:46.994344    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:47.282858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:47.282780    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64942300-c73a-492a-a57a-c48930e12985" path="/var/lib/kubelet/pods/64942300-c73a-492a-a57a-c48930e12985/volumes"
Oct 30 06:11:48.573525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:48.573484    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:49.372382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.372348    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"]
Oct 30 06:11:49.373009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.372991    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b66db160-1292-4851-b8cc-4e53182b0acd" containerName="revoke-keys"
Oct 30 06:11:49.373009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.373008    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66db160-1292-4851-b8cc-4e53182b0acd" containerName="revoke-keys"
Oct 30 06:11:49.373219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.373208    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66db160-1292-4851-b8cc-4e53182b0acd" containerName="revoke-keys"
Oct 30 06:11:49.377214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.377194    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.390393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.390367    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-0b481b-openshift-default-dockercfg-nrg7r\""
Oct 30 06:11:49.406458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.406429    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"]
Oct 30 06:11:49.410161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.410138    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.410371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.410339    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzprr\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-kube-api-access-nzprr\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.410445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.410399    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.410524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.410499    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.410631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.410612    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.410715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.410642    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.410715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.410706    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.410910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.410891    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.411070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.411047    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.511998    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512034    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512078    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512094    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512127    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512259    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512573 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512349    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512573 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512382    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512573 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512404    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzprr\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-kube-api-access-nzprr\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512573 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512475    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512573 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512530    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512690    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.512995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.512977    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.513402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.513373    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.514594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.514570    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.515132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.515111    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.533519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.533499    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.533891 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.533872    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzprr\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-kube-api-access-nzprr\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:49.691477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:49.691443    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:50.016206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:50.016160    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"]
Oct 30 06:11:50.017282 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:11:50.017250    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba977e9_e030_40a0_a519_9b3fc81b1a16.slice/crio-c48783607ef5a748b68b9223fa666e90eaa9a2d289934c745644a5f278b5a77a WatchSource:0}: Error finding container c48783607ef5a748b68b9223fa666e90eaa9a2d289934c745644a5f278b5a77a: Status 404 returned error can't find the container with id c48783607ef5a748b68b9223fa666e90eaa9a2d289934c745644a5f278b5a77a
Oct 30 06:11:50.020024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:50.019983    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:11:50.020109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:50.020060    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:11:50.020109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:50.020089    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:11:50.081451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:50.081423    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:50.697218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:50.697150    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" event={"ID":"9ba977e9-e030-40a0-a519-9b3fc81b1a16","Type":"ContainerStarted","Data":"ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501"}
Oct 30 06:11:50.697631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:50.697241    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" event={"ID":"9ba977e9-e030-40a0-a519-9b3fc81b1a16","Type":"ContainerStarted","Data":"c48783607ef5a748b68b9223fa666e90eaa9a2d289934c745644a5f278b5a77a"}
Oct 30 06:11:50.767234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:50.763683    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" podStartSLOduration=1.763661696 podStartE2EDuration="1.763661696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:50.748504691 +0000 UTC m=+2425.579541908" watchObservedRunningTime="2026-10-30 06:11:50.763661696 +0000 UTC m=+2425.594698915"
Oct 30 06:11:51.326976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.326943    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"]
Oct 30 06:11:51.331376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.331355    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.348034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.346714    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b4ca14\"/\"openshift-service-ca.crt\""
Oct 30 06:11:51.348034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.346952    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b4ca14\"/\"kube-root-ca.crt\""
Oct 30 06:11:51.383241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.374445    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b4ca14\"/\"auth-test-model-b4ca14-kserve-self-signed-certs\""
Oct 30 06:11:51.383241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.381337    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b4ca14\"/\"default-dockercfg-vcwzk\""
Oct 30 06:11:51.385222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.385149    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"]
Oct 30 06:11:51.417140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.417103    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:51.438540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.438504    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-dshm\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.438728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.438628    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-model-cache\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.438728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.438678    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4k2\" (UniqueName: \"kubernetes.io/projected/082b94fb-4e5e-4739-8616-85ad30f74065-kube-api-access-jf4k2\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.438849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.438751    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082b94fb-4e5e-4739-8616-85ad30f74065-tls-certs\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.438849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.438789    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-tmp-dir\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.438961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.438847    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-home\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.539984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.539951    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-dshm\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.540220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.540030    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-model-cache\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.540220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.540070    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4k2\" (UniqueName: \"kubernetes.io/projected/082b94fb-4e5e-4739-8616-85ad30f74065-kube-api-access-jf4k2\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.540220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.540135    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082b94fb-4e5e-4739-8616-85ad30f74065-tls-certs\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.540220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.540168    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-tmp-dir\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.540475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.540248    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-home\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.540533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.540518    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-model-cache\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.540792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.540718    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-home\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.540792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.540739    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-tmp-dir\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.543020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.542991    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-dshm\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.543131 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.543053    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082b94fb-4e5e-4739-8616-85ad30f74065-tls-certs\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.586253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.583884    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4k2\" (UniqueName: \"kubernetes.io/projected/082b94fb-4e5e-4739-8616-85ad30f74065-kube-api-access-jf4k2\") pod \"auth-test-model-b4ca14-kserve-76d944d8f-tm84p\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") " pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.653207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.652439    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:51.704598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.692047    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:51.704598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.697250    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 06:11:51.704598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.697302    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 06:11:51.896268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:51.881755    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:52.084204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:52.083594    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"]
Oct 30 06:11:52.092762 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:11:52.092734    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod082b94fb_4e5e_4739_8616_85ad30f74065.slice/crio-c493cc8d71ac4fce84b1145355707d2fe0e50fe6d3680a503854b6054c3ca5e9 WatchSource:0}: Error finding container c493cc8d71ac4fce84b1145355707d2fe0e50fe6d3680a503854b6054c3ca5e9: Status 404 returned error can't find the container with id c493cc8d71ac4fce84b1145355707d2fe0e50fe6d3680a503854b6054c3ca5e9
Oct 30 06:11:52.691762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:52.691727    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 06:11:52.691956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:52.691782    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 06:11:52.722528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:52.722489    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p" event={"ID":"082b94fb-4e5e-4739-8616-85ad30f74065","Type":"ContainerStarted","Data":"2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91"}
Oct 30 06:11:52.722528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:52.722533    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p" event={"ID":"082b94fb-4e5e-4739-8616-85ad30f74065","Type":"ContainerStarted","Data":"c493cc8d71ac4fce84b1145355707d2fe0e50fe6d3680a503854b6054c3ca5e9"}
Oct 30 06:11:53.482841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:53.482806    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:53.653790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:53.653753    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:11:53.691847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:53.691815    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 06:11:53.692028 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:53.691868    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 06:11:54.710245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:54.702274    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 06:11:54.710245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:54.702332    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 06:11:54.710245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:54.703481    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p" podStartSLOduration=3.70346025 podStartE2EDuration="3.70346025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:52.781062359 +0000 UTC m=+2427.612099577" watchObservedRunningTime="2026-10-30 06:11:54.70346025 +0000 UTC m=+2429.534497468"
Oct 30 06:11:54.714205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:54.713876    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"]
Oct 30 06:11:54.738759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:54.738719    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:55.291085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:55.291051    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:55.692104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:55.692061    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 06:11:55.692318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:55.692121    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 06:11:56.696610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:56.696569    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:56.697142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:56.697077    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:56.697905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:56.697884    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:11:56.910533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:56.910498    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:57.986320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:57.986272    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"]
Oct 30 06:11:58.061658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:58.061622    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:11:58.676410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:11:58.676373    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:00.440896 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:00.440855    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:00.585062 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:00.585022    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:00.735808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:00.735730    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:01.183590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:01.183550    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"]
Oct 30 06:12:01.258285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:01.258249    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:01.653447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:01.653397    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:12:01.672322 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:01.672287    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:12:01.825309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:01.825264    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:12:02.163440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:02.163406    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:02.374472 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:12:02.374438    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:12:03.771941 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:03.771905    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:05.527392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:05.527345    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:07.077440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:07.077398    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:08.641438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:08.641381    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:08.790195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:08.790148    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:09.924764 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:09.924691    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8" podUID="762fb9b3-a800-478d-bb49-9180d8ece9be" containerName="main" containerID="cri-o://4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00" gracePeriod=45
Oct 30 06:12:10.211310 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.211279    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:12:10.315793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.315747    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-home\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-home\") pod \"762fb9b3-a800-478d-bb49-9180d8ece9be\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") "
Oct 30 06:12:10.316002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.315874    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762fb9b3-a800-478d-bb49-9180d8ece9be-kube-api-access-s5zn5\" (UniqueName: \"kubernetes.io/projected/762fb9b3-a800-478d-bb49-9180d8ece9be-kube-api-access-s5zn5\") pod \"762fb9b3-a800-478d-bb49-9180d8ece9be\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") "
Oct 30 06:12:10.316002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.315965    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-model-cache\") pod \"762fb9b3-a800-478d-bb49-9180d8ece9be\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") "
Oct 30 06:12:10.316128 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316043    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-home" pod "762fb9b3-a800-478d-bb49-9180d8ece9be" (UID: "762fb9b3-a800-478d-bb49-9180d8ece9be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:10.316128 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316111    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-tmp-dir\") pod \"762fb9b3-a800-478d-bb49-9180d8ece9be\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") "
Oct 30 06:12:10.316361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316166    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762fb9b3-a800-478d-bb49-9180d8ece9be-tls-certs\" (UniqueName: \"kubernetes.io/secret/762fb9b3-a800-478d-bb49-9180d8ece9be-tls-certs\") pod \"762fb9b3-a800-478d-bb49-9180d8ece9be\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") "
Oct 30 06:12:10.316361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316230    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-dshm\") pod \"762fb9b3-a800-478d-bb49-9180d8ece9be\" (UID: \"762fb9b3-a800-478d-bb49-9180d8ece9be\") "
Oct 30 06:12:10.316361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316284    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-model-cache" pod "762fb9b3-a800-478d-bb49-9180d8ece9be" (UID: "762fb9b3-a800-478d-bb49-9180d8ece9be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:10.316571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316404    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-tmp-dir" pod "762fb9b3-a800-478d-bb49-9180d8ece9be" (UID: "762fb9b3-a800-478d-bb49-9180d8ece9be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:10.316828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316807    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:10.316909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316833    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:10.316909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.316846    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:10.318497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.318466    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-dshm" pod "762fb9b3-a800-478d-bb49-9180d8ece9be" (UID: "762fb9b3-a800-478d-bb49-9180d8ece9be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:10.318623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.318500    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762fb9b3-a800-478d-bb49-9180d8ece9be-kube-api-access-s5zn5" pod "762fb9b3-a800-478d-bb49-9180d8ece9be" (UID: "762fb9b3-a800-478d-bb49-9180d8ece9be"). InnerVolumeSpecName "kube-api-access-s5zn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:12:10.318623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.318537    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762fb9b3-a800-478d-bb49-9180d8ece9be-tls-certs" pod "762fb9b3-a800-478d-bb49-9180d8ece9be" (UID: "762fb9b3-a800-478d-bb49-9180d8ece9be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:12:10.418746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.418712    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762fb9b3-a800-478d-bb49-9180d8ece9be-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:10.418746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.418746    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762fb9b3-a800-478d-bb49-9180d8ece9be-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:10.418934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.418763    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5zn5\" (UniqueName: \"kubernetes.io/projected/762fb9b3-a800-478d-bb49-9180d8ece9be-kube-api-access-s5zn5\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:10.513508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.513353    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:10.884664 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.884630    5250 generic.go:356] "Generic (PLEG): container finished" podID="762fb9b3-a800-478d-bb49-9180d8ece9be" containerID="4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00" exitCode=0
Oct 30 06:12:10.884868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.884677    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"
Oct 30 06:12:10.884868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.884713    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8" event={"ID":"762fb9b3-a800-478d-bb49-9180d8ece9be","Type":"ContainerDied","Data":"4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00"}
Oct 30 06:12:10.884868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.884767    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8" event={"ID":"762fb9b3-a800-478d-bb49-9180d8ece9be","Type":"ContainerDied","Data":"13ea19bc0671210ec882dcf7b9e1387439beec6e659cc12741d42c45d2b791e0"}
Oct 30 06:12:10.884868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.884790    5250 scope.go:122] "RemoveContainer" containerID="4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00"
Oct 30 06:12:10.897107 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.897081    5250 scope.go:122] "RemoveContainer" containerID="4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00"
Oct 30 06:12:10.897394 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:12:10.897370    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00\": container with ID starting with 4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00 not found: ID does not exist" containerID="4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00"
Oct 30 06:12:10.897486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.897406    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00"} err="failed to get container status \"4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00\": rpc error: code = NotFound desc = could not find container \"4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00\": container with ID starting with 4d09dfbc43d6b1675a1a6868450b8bf2dad0adefa8a8301e86fcdc30b9e3ca00 not found: ID does not exist"
Oct 30 06:12:10.932921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.932894    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"]
Oct 30 06:12:10.943765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:10.943736    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/sub-test-model-452a59-kserve-676b9d4b8f-9jng8"]
Oct 30 06:12:11.283029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:11.282952    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762fb9b3-a800-478d-bb49-9180d8ece9be" path="/var/lib/kubelet/pods/762fb9b3-a800-478d-bb49-9180d8ece9be/volumes"
Oct 30 06:12:12.034427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:12.034392    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:13.013094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.013038    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk" podUID="6e678012-1dd4-48c5-b76d-d325bc7779b2" containerName="main" containerID="cri-o://7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f" gracePeriod=45
Oct 30 06:12:13.321586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.321545    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:12:13.378335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.375791    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:13.474033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.473976    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-dshm\") pod \"6e678012-1dd4-48c5-b76d-d325bc7779b2\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") "
Oct 30 06:12:13.474825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.474800    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e678012-1dd4-48c5-b76d-d325bc7779b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e678012-1dd4-48c5-b76d-d325bc7779b2-tls-certs\") pod \"6e678012-1dd4-48c5-b76d-d325bc7779b2\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") "
Oct 30 06:12:13.475500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.475469    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e678012-1dd4-48c5-b76d-d325bc7779b2-kube-api-access-5kdhv\" (UniqueName: \"kubernetes.io/projected/6e678012-1dd4-48c5-b76d-d325bc7779b2-kube-api-access-5kdhv\") pod \"6e678012-1dd4-48c5-b76d-d325bc7779b2\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") "
Oct 30 06:12:13.476069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.476049    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-model-cache\") pod \"6e678012-1dd4-48c5-b76d-d325bc7779b2\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") "
Oct 30 06:12:13.476258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.476229    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-home\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-home\") pod \"6e678012-1dd4-48c5-b76d-d325bc7779b2\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") "
Oct 30 06:12:13.476508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.476491    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-tmp-dir\") pod \"6e678012-1dd4-48c5-b76d-d325bc7779b2\" (UID: \"6e678012-1dd4-48c5-b76d-d325bc7779b2\") "
Oct 30 06:12:13.477939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.477908    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-tmp-dir" pod "6e678012-1dd4-48c5-b76d-d325bc7779b2" (UID: "6e678012-1dd4-48c5-b76d-d325bc7779b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:13.478319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.478300    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-model-cache" pod "6e678012-1dd4-48c5-b76d-d325bc7779b2" (UID: "6e678012-1dd4-48c5-b76d-d325bc7779b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:13.478641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.478624    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-home" pod "6e678012-1dd4-48c5-b76d-d325bc7779b2" (UID: "6e678012-1dd4-48c5-b76d-d325bc7779b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:13.497352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.495365    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e678012-1dd4-48c5-b76d-d325bc7779b2-kube-api-access-5kdhv" pod "6e678012-1dd4-48c5-b76d-d325bc7779b2" (UID: "6e678012-1dd4-48c5-b76d-d325bc7779b2"). InnerVolumeSpecName "kube-api-access-5kdhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:12:13.497352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.495634    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-dshm" pod "6e678012-1dd4-48c5-b76d-d325bc7779b2" (UID: "6e678012-1dd4-48c5-b76d-d325bc7779b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:13.497352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.496248    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e678012-1dd4-48c5-b76d-d325bc7779b2-tls-certs" pod "6e678012-1dd4-48c5-b76d-d325bc7779b2" (UID: "6e678012-1dd4-48c5-b76d-d325bc7779b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:12:13.580227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.580100    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:13.580227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.580142    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:13.580227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.580195    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:13.580227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.580208    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e678012-1dd4-48c5-b76d-d325bc7779b2-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:13.580227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.580220    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e678012-1dd4-48c5-b76d-d325bc7779b2-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:13.580545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.580235    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kdhv\" (UniqueName: \"kubernetes.io/projected/6e678012-1dd4-48c5-b76d-d325bc7779b2-kube-api-access-5kdhv\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:13.644692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.644652    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:13.910401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.910372    5250 generic.go:356] "Generic (PLEG): container finished" podID="6e678012-1dd4-48c5-b76d-d325bc7779b2" containerID="7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f" exitCode=0
Oct 30 06:12:13.910603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.910407    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk" event={"ID":"6e678012-1dd4-48c5-b76d-d325bc7779b2","Type":"ContainerDied","Data":"7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f"}
Oct 30 06:12:13.910603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.910437    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"
Oct 30 06:12:13.910603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.910457    5250 scope.go:122] "RemoveContainer" containerID="7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f"
Oct 30 06:12:13.910603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.910438    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk" event={"ID":"6e678012-1dd4-48c5-b76d-d325bc7779b2","Type":"ContainerDied","Data":"e2a7a6c918f646cb7e8d5b340dbf08f7749bcc9a31d8d85e7273e64d6369d859"}
Oct 30 06:12:13.922233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.922172    5250 scope.go:122] "RemoveContainer" containerID="7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f"
Oct 30 06:12:13.922493 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:12:13.922470    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f\": container with ID starting with 7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f not found: ID does not exist" containerID="7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f"
Oct 30 06:12:13.922549 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.922501    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f"} err="failed to get container status \"7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f\": rpc error: code = NotFound desc = could not find container \"7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f\": container with ID starting with 7fc22e23e6747831fea1ef54b3661e3e8dd1b530ecc7b836ba129aa6b705b21f not found: ID does not exist"
Oct 30 06:12:13.960213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.960154    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"]
Oct 30 06:12:13.970810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:13.970782    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk"]
Oct 30 06:12:14.091819 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.091786    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:14.245280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.245172    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-080391/auth-test-model-080391-kserve-fd748b7f7-ngvqk" podUID="6e678012-1dd4-48c5-b76d-d325bc7779b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:12:14.445213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.443150    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"]
Oct 30 06:12:14.445213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.444247    5250 cpu_manager.go:414] "containerMap: removing container" podUID="762fb9b3-a800-478d-bb49-9180d8ece9be" containerName="main"
Oct 30 06:12:14.445213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.444268    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762fb9b3-a800-478d-bb49-9180d8ece9be" containerName="main"
Oct 30 06:12:14.445213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.444343    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6e678012-1dd4-48c5-b76d-d325bc7779b2" containerName="main"
Oct 30 06:12:14.445213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.444353    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e678012-1dd4-48c5-b76d-d325bc7779b2" containerName="main"
Oct 30 06:12:14.445213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.444549    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e678012-1dd4-48c5-b76d-d325bc7779b2" containerName="main"
Oct 30 06:12:14.445213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.444575    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="762fb9b3-a800-478d-bb49-9180d8ece9be" containerName="main"
Oct 30 06:12:14.451319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.451294    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.469725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.469695    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"]
Oct 30 06:12:14.474540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.474517    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-36ae27-openshift-default-dockercfg-xvkqn\""
Oct 30 06:12:14.493516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.493479    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-token\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.493688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.493601    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-podinfo\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.493779 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.493731    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-envoy\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.493840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.493811    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-certs\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.493903 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.493859    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-credential-socket\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.493903 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.493891    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-socket\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.494011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.493916    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-data\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.494011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.493948    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gtg\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-kube-api-access-h6gtg\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.494011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.494002    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f242f-1a7a-4a18-93ae-66d06f77f88f-istiod-ca-cert\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.530877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.530804    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601160    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-envoy\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601323    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-certs\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601376    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-credential-socket\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601409    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-socket\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601437    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-data\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601471    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gtg\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-kube-api-access-h6gtg\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601525    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f242f-1a7a-4a18-93ae-66d06f77f88f-istiod-ca-cert\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601592    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-token\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601696    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-podinfo\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.601963    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-certs\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.602228    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-credential-socket\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.602689    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-socket\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.602896    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f242f-1a7a-4a18-93ae-66d06f77f88f-istiod-ca-cert\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.604237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.602924    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-data\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.645209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.645030    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-envoy\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.660202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.656784    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-token\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.660202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.656869    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gtg\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-kube-api-access-h6gtg\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.696209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.694254    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-podinfo\") pod \"e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") " pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:14.697205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.697043    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:14.770507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:14.770467    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:15.139201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:15.135785    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"]
Oct 30 06:12:15.146077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:15.146040    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:15.146192 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:15.146113    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:15.146192 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:15.146148    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:15.347204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:15.340658    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e678012-1dd4-48c5-b76d-d325bc7779b2" path="/var/lib/kubelet/pods/6e678012-1dd4-48c5-b76d-d325bc7779b2/volumes"
Oct 30 06:12:16.028212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.026547    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" event={"ID":"966f242f-1a7a-4a18-93ae-66d06f77f88f","Type":"ContainerStarted","Data":"f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b"}
Oct 30 06:12:16.028212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.026626    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" event={"ID":"966f242f-1a7a-4a18-93ae-66d06f77f88f","Type":"ContainerStarted","Data":"ff73f47485937d23daf5cd44291f2587e83de79135cc9aff9e116f33ed6ac4c3"}
Oct 30 06:12:16.091955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.091899    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podStartSLOduration=2.091878701 podStartE2EDuration="2.091878701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:16.084683937 +0000 UTC m=+2450.915721149" watchObservedRunningTime="2026-10-30 06:12:16.091878701 +0000 UTC m=+2450.922915914"
Oct 30 06:12:16.199257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.193432    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:16.221352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.220713    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4" podUID="b9e94010-987c-4a49-bb42-3a1dd7a495da" containerName="main" containerID="cri-o://ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8" gracePeriod=45
Oct 30 06:12:16.438763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.438716    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4" podUID="b9e94010-987c-4a49-bb42-3a1dd7a495da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.185:8000/ready\": dial tcp 10.133.0.185:8000: connect: connection refused"
Oct 30 06:12:16.730799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.730771    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:12:16.780205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.779228    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:16.790434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.790048    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:16.856876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.856780    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-home\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-home\") pod \"b9e94010-987c-4a49-bb42-3a1dd7a495da\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") "
Oct 30 06:12:16.856876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.856848    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e94010-987c-4a49-bb42-3a1dd7a495da-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e94010-987c-4a49-bb42-3a1dd7a495da-tls-certs\") pod \"b9e94010-987c-4a49-bb42-3a1dd7a495da\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") "
Oct 30 06:12:16.857133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.856888    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-dshm\") pod \"b9e94010-987c-4a49-bb42-3a1dd7a495da\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") "
Oct 30 06:12:16.857458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.857407    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-home" pod "b9e94010-987c-4a49-bb42-3a1dd7a495da" (UID: "b9e94010-987c-4a49-bb42-3a1dd7a495da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:16.857550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.857464    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-model-cache" pod "b9e94010-987c-4a49-bb42-3a1dd7a495da" (UID: "b9e94010-987c-4a49-bb42-3a1dd7a495da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:16.857667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.857643    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-model-cache\") pod \"b9e94010-987c-4a49-bb42-3a1dd7a495da\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") "
Oct 30 06:12:16.857858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.857838    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e94010-987c-4a49-bb42-3a1dd7a495da-kube-api-access-fqs25\" (UniqueName: \"kubernetes.io/projected/b9e94010-987c-4a49-bb42-3a1dd7a495da-kube-api-access-fqs25\") pod \"b9e94010-987c-4a49-bb42-3a1dd7a495da\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") "
Oct 30 06:12:16.858111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.858092    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-tmp-dir\") pod \"b9e94010-987c-4a49-bb42-3a1dd7a495da\" (UID: \"b9e94010-987c-4a49-bb42-3a1dd7a495da\") "
Oct 30 06:12:16.858442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.858416    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-tmp-dir" pod "b9e94010-987c-4a49-bb42-3a1dd7a495da" (UID: "b9e94010-987c-4a49-bb42-3a1dd7a495da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:16.859070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.859045    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:16.859224 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.859209    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:16.859330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.859318    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:16.861034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.860984    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-dshm" pod "b9e94010-987c-4a49-bb42-3a1dd7a495da" (UID: "b9e94010-987c-4a49-bb42-3a1dd7a495da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:12:16.861145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.861033    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e94010-987c-4a49-bb42-3a1dd7a495da-tls-certs" pod "b9e94010-987c-4a49-bb42-3a1dd7a495da" (UID: "b9e94010-987c-4a49-bb42-3a1dd7a495da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:12:16.863229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.863171    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e94010-987c-4a49-bb42-3a1dd7a495da-kube-api-access-fqs25" pod "b9e94010-987c-4a49-bb42-3a1dd7a495da" (UID: "b9e94010-987c-4a49-bb42-3a1dd7a495da"). InnerVolumeSpecName "kube-api-access-fqs25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:12:16.960511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.960475    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqs25\" (UniqueName: \"kubernetes.io/projected/b9e94010-987c-4a49-bb42-3a1dd7a495da-kube-api-access-fqs25\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:16.960511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.960509    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e94010-987c-4a49-bb42-3a1dd7a495da-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:16.960760 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:16.960523    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e94010-987c-4a49-bb42-3a1dd7a495da-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:17.036613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.036574    5250 generic.go:356] "Generic (PLEG): container finished" podID="b9e94010-987c-4a49-bb42-3a1dd7a495da" containerID="ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8" exitCode=0
Oct 30 06:12:17.037052 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.036635    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"
Oct 30 06:12:17.037131 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.036622    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4" event={"ID":"b9e94010-987c-4a49-bb42-3a1dd7a495da","Type":"ContainerDied","Data":"ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8"}
Oct 30 06:12:17.037238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.037163    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4" event={"ID":"b9e94010-987c-4a49-bb42-3a1dd7a495da","Type":"ContainerDied","Data":"e896232e288d68900214595306b41a1fefee7870d6b1568cbbb294467e7db756"}
Oct 30 06:12:17.037238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.037232    5250 scope.go:122] "RemoveContainer" containerID="ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8"
Oct 30 06:12:17.037693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.037671    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:17.038751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.038731    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:12:17.051754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.051734    5250 scope.go:122] "RemoveContainer" containerID="ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8"
Oct 30 06:12:17.052096 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:12:17.052068    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8\": container with ID starting with ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8 not found: ID does not exist" containerID="ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8"
Oct 30 06:12:17.052206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.052104    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8"} err="failed to get container status \"ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8\": rpc error: code = NotFound desc = could not find container \"ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8\": container with ID starting with ce06dfac0bae2e8ebf9a76e576730d2fde73733e5dc543a446a1cd0d0760f3f8 not found: ID does not exist"
Oct 30 06:12:17.141250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.137973    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"]
Oct 30 06:12:17.143410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.143383    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-1b4b7a/rate-test-model-1b4b7a-kserve-76796bfc9c-5rnx4"]
Oct 30 06:12:17.283634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.283597    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e94010-987c-4a49-bb42-3a1dd7a495da" path="/var/lib/kubelet/pods/b9e94010-987c-4a49-bb42-3a1dd7a495da/volumes"
Oct 30 06:12:17.911211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:17.908858    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:18.616926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.616893    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:18.633303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.633273    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"]
Oct 30 06:12:18.634284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.634256    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b9e94010-987c-4a49-bb42-3a1dd7a495da" containerName="main"
Oct 30 06:12:18.634284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.634283    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e94010-987c-4a49-bb42-3a1dd7a495da" containerName="main"
Oct 30 06:12:18.634510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.634482    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e94010-987c-4a49-bb42-3a1dd7a495da" containerName="main"
Oct 30 06:12:18.639977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.639955    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.654035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.654009    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"]
Oct 30 06:12:18.655826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.655799    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-c2be21\"/\"default-dockercfg-x84qj\""
Oct 30 06:12:18.667469 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.667438    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-c2be21\"/\"openshift-service-ca.crt\""
Oct 30 06:12:18.667469 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.667452    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-c2be21\"/\"kube-root-ca.crt\""
Oct 30 06:12:18.735897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.735868    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-c2be21\"/\"sub-test-model-c2be21-kserve-self-signed-certs\""
Oct 30 06:12:18.785161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.785123    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-dshm\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.785358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.785193    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr7px\" (UniqueName: \"kubernetes.io/projected/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-kube-api-access-lr7px\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.785450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.785387    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tmp-dir\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.785521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.785496    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-model-cache\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.785639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.785618    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-home\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.785698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.785664    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tls-certs\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.789192 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.789153    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.886701    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-model-cache\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.886840    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-home\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.886891    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tls-certs\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.887029    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-dshm\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.887068    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7px\" (UniqueName: \"kubernetes.io/projected/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-kube-api-access-lr7px\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.887132    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tmp-dir\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.887671    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tmp-dir\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.887935    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-model-cache\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.890207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.888199    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-home\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.925211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.924334    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-dshm\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.925211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.925048    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7px\" (UniqueName: \"kubernetes.io/projected/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-kube-api-access-lr7px\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.954204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.943795    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tls-certs\") pod \"sub-test-model-c2be21-kserve-66d6988775-zkp75\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") " pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:18.977203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:18.958504    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:19.429479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:19.429445    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"]
Oct 30 06:12:19.430720 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:12:19.430691    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd76cc8a_8336_4d83_a616_f53b4a6fd4dc.slice/crio-72d82ade8801ec3438fd8d99a7080b43c5e8e85729206a52fa857a1817026c32 WatchSource:0}: Error finding container 72d82ade8801ec3438fd8d99a7080b43c5e8e85729206a52fa857a1817026c32: Status 404 returned error can't find the container with id 72d82ade8801ec3438fd8d99a7080b43c5e8e85729206a52fa857a1817026c32
Oct 30 06:12:19.930016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:19.929983    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:20.073212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.072827    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75" event={"ID":"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc","Type":"ContainerStarted","Data":"791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a"}
Oct 30 06:12:20.073212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.072873    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75" event={"ID":"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc","Type":"ContainerStarted","Data":"72d82ade8801ec3438fd8d99a7080b43c5e8e85729206a52fa857a1817026c32"}
Oct 30 06:12:20.124594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.124520    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75" podStartSLOduration=2.124495595 podStartE2EDuration="2.124495595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:20.121754047 +0000 UTC m=+2454.952791264" watchObservedRunningTime="2026-10-30 06:12:20.124495595 +0000 UTC m=+2454.955532804"
Oct 30 06:12:20.325159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.325012    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:20.618925 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.618845    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:20.965240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.959786    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:20.965240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.962355    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"]
Oct 30 06:12:20.967907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.967884    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.000654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:20.999991    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"]
Oct 30 06:12:21.058634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.058595    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:21.073472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.073440    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ebe61e-openshift-default-dockercfg-5czlr\""
Oct 30 06:12:21.135297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.135263    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"]
Oct 30 06:12:21.140758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.140729    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:21.142931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.142903    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4jwj\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-kube-api-access-l4jwj\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.143139 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.143116    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-socket\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.144287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.143406    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-envoy\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.144287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.143449    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-podinfo\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.144287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.143490    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-token\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.144287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.143542    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-certs\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.144287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.143676    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istiod-ca-cert\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.144287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.143764    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-credential-socket\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.144287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.143799    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-data\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.166581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.166556    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"]
Oct 30 06:12:21.173495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.173471    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-36ae27-dockercfg-npxdj\""
Oct 30 06:12:21.225999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.225921    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:21.244711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.244678    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-36ae27\""
Oct 30 06:12:21.245270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245247    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istiod-ca-cert\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245385 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245361    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-credential-socket\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245399    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-data\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245443    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d350f3ef-5d12-4021-9fa5-88e397430ecf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc\" (UID: \"d350f3ef-5d12-4021-9fa5-88e397430ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:21.245586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245567    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4jwj\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-kube-api-access-l4jwj\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245601    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-socket\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245650    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-envoy\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245678    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-podinfo\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245715    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2v8\" (UniqueName: \"kubernetes.io/projected/d350f3ef-5d12-4021-9fa5-88e397430ecf-kube-api-access-gd2v8\") pod \"payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc\" (UID: \"d350f3ef-5d12-4021-9fa5-88e397430ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:21.245825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245749    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-token\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245804    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-certs\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.245825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.245813    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-data\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.246059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.246031    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istiod-ca-cert\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.246456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.246379    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-certs\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.246456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.246388    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-credential-socket\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.246880 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.246854    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-socket\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.248300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.248281    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-envoy\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.249254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.249235    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-podinfo\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.268086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.268062    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-token\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.268929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.268908    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4jwj\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-kube-api-access-l4jwj\") pod \"e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") " pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.296073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.296042    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:21.372268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.371996    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:21.408296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.398849    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d350f3ef-5d12-4021-9fa5-88e397430ecf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc\" (UID: \"d350f3ef-5d12-4021-9fa5-88e397430ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:21.408296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.399024    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2v8\" (UniqueName: \"kubernetes.io/projected/d350f3ef-5d12-4021-9fa5-88e397430ecf-kube-api-access-gd2v8\") pod \"payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc\" (UID: \"d350f3ef-5d12-4021-9fa5-88e397430ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:21.408296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.400433    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d350f3ef-5d12-4021-9fa5-88e397430ecf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc\" (UID: \"d350f3ef-5d12-4021-9fa5-88e397430ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:21.446206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.445162    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2v8\" (UniqueName: \"kubernetes.io/projected/d350f3ef-5d12-4021-9fa5-88e397430ecf-kube-api-access-gd2v8\") pod \"payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc\" (UID: \"d350f3ef-5d12-4021-9fa5-88e397430ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:21.481276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.472436    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:21.806870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.806840    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"]
Oct 30 06:12:21.809446 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:12:21.809417    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f40cc9b_5c8e_414e_8b57_223c8b18b40f.slice/crio-16de5b094d0fa0ac459376507549134e08d52692a6058a7078f82fbfa28b3f75 WatchSource:0}: Error finding container 16de5b094d0fa0ac459376507549134e08d52692a6058a7078f82fbfa28b3f75: Status 404 returned error can't find the container with id 16de5b094d0fa0ac459376507549134e08d52692a6058a7078f82fbfa28b3f75
Oct 30 06:12:21.813042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.813002    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:21.813149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.813089    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:21.813149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.813129    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:21.856346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:21.856317    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"]
Oct 30 06:12:21.858764 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:12:21.858739    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd350f3ef_5d12_4021_9fa5_88e397430ecf.slice/crio-508105d5fa4984291ddcd0154a7aba324b62581edd4fa4e8f15ea7dd6939d837 WatchSource:0}: Error finding container 508105d5fa4984291ddcd0154a7aba324b62581edd4fa4e8f15ea7dd6939d837: Status 404 returned error can't find the container with id 508105d5fa4984291ddcd0154a7aba324b62581edd4fa4e8f15ea7dd6939d837
Oct 30 06:12:22.015231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.015161    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:22.098890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.098734    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc" event={"ID":"d350f3ef-5d12-4021-9fa5-88e397430ecf","Type":"ContainerStarted","Data":"70a9ffc79ef512015e06912dbd7229394bd5316807b18ce786776a43030f67cf"}
Oct 30 06:12:22.098890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.098804    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc" event={"ID":"d350f3ef-5d12-4021-9fa5-88e397430ecf","Type":"ContainerStarted","Data":"508105d5fa4984291ddcd0154a7aba324b62581edd4fa4e8f15ea7dd6939d837"}
Oct 30 06:12:22.099702 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.099633    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:22.101532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.101471    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" event={"ID":"2f40cc9b-5c8e-414e-8b57-223c8b18b40f","Type":"ContainerStarted","Data":"e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e"}
Oct 30 06:12:22.101532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.101507    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" event={"ID":"2f40cc9b-5c8e-414e-8b57-223c8b18b40f","Type":"ContainerStarted","Data":"16de5b094d0fa0ac459376507549134e08d52692a6058a7078f82fbfa28b3f75"}
Oct 30 06:12:22.143112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.142664    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc" podStartSLOduration=1.142645556 podStartE2EDuration="1.142645556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:22.136612522 +0000 UTC m=+2456.967649740" watchObservedRunningTime="2026-10-30 06:12:22.142645556 +0000 UTC m=+2456.973682775"
Oct 30 06:12:22.183479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.183413    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podStartSLOduration=2.18339514 podStartE2EDuration="2.18339514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:22.179376411 +0000 UTC m=+2457.010413630" watchObservedRunningTime="2026-10-30 06:12:22.18339514 +0000 UTC m=+2457.014432360"
Oct 30 06:12:22.303632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.303594    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:22.309339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.309302    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 06:12:22.309627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:22.309597    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 06:12:23.028318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:23.028281    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:23.302042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:23.301968    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:23.800516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:23.800461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:24.117824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:24.117737    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:24.118660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:24.118639    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:12:24.648620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:24.648589    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:24.798136 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:24.798103    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:25.680790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:25.680755    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:26.239498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.239453    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:26.308753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.308713    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"]
Oct 30 06:12:26.314774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.314748    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:26.335672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.335642    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"]
Oct 30 06:12:26.341111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.341091    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ebe61e-dockercfg-9sh4d\""
Oct 30 06:12:26.375689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.375651    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c95pb\" (UniqueName: \"kubernetes.io/projected/017b8f8e-f88d-48da-b101-7bd8721647aa-kube-api-access-c95pb\") pod \"payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf\" (UID: \"017b8f8e-f88d-48da-b101-7bd8721647aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:26.375837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.375792    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b8f8e-f88d-48da-b101-7bd8721647aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf\" (UID: \"017b8f8e-f88d-48da-b101-7bd8721647aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:26.382661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.382637    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:26.418999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.418976    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ebe61e\""
Oct 30 06:12:26.484016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.483976    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c95pb\" (UniqueName: \"kubernetes.io/projected/017b8f8e-f88d-48da-b101-7bd8721647aa-kube-api-access-c95pb\") pod \"payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf\" (UID: \"017b8f8e-f88d-48da-b101-7bd8721647aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:26.484795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.484766    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b8f8e-f88d-48da-b101-7bd8721647aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf\" (UID: \"017b8f8e-f88d-48da-b101-7bd8721647aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:26.510629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.485784    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b8f8e-f88d-48da-b101-7bd8721647aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf\" (UID: \"017b8f8e-f88d-48da-b101-7bd8721647aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:26.531638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.530346    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c95pb\" (UniqueName: \"kubernetes.io/projected/017b8f8e-f88d-48da-b101-7bd8721647aa-kube-api-access-c95pb\") pod \"payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf\" (UID: \"017b8f8e-f88d-48da-b101-7bd8721647aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:26.635384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.635341    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:26.944699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:26.944675    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"]
Oct 30 06:12:26.945598 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:12:26.945565    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017b8f8e_f88d_48da_b101_7bd8721647aa.slice/crio-4be5e97338f3227078c2328d8b0d71b80da75ced659d4e6504bea8b2b12a6a3a WatchSource:0}: Error finding container 4be5e97338f3227078c2328d8b0d71b80da75ced659d4e6504bea8b2b12a6a3a: Status 404 returned error can't find the container with id 4be5e97338f3227078c2328d8b0d71b80da75ced659d4e6504bea8b2b12a6a3a
Oct 30 06:12:27.145998 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:27.145966    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf" event={"ID":"017b8f8e-f88d-48da-b101-7bd8721647aa","Type":"ContainerStarted","Data":"604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee"}
Oct 30 06:12:27.146236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:27.146025    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf" event={"ID":"017b8f8e-f88d-48da-b101-7bd8721647aa","Type":"ContainerStarted","Data":"4be5e97338f3227078c2328d8b0d71b80da75ced659d4e6504bea8b2b12a6a3a"}
Oct 30 06:12:27.146236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:27.146062    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:27.185614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:27.185564    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf" podStartSLOduration=1.1855503490000001 podStartE2EDuration="1.185550349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:27.182232313 +0000 UTC m=+2462.013269527" watchObservedRunningTime="2026-10-30 06:12:27.185550349 +0000 UTC m=+2462.016587605"
Oct 30 06:12:27.411777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:27.411747    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:28.029977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:28.029944    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"]
Oct 30 06:12:28.040880 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:28.040848    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7c3554-zkckq"]
Oct 30 06:12:28.958950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:28.958917    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:28.972038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:28.972004    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:29.174665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:29.174635    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:12:29.275661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:29.275561    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:29.284065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:29.284035    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ae2a46-476b-4396-89e3-9dd3d6bffb65" path="/var/lib/kubelet/pods/06ae2a46-476b-4396-89e3-9dd3d6bffb65/volumes"
Oct 30 06:12:31.043236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.043200    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:31.632053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.632022    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"]
Oct 30 06:12:31.638066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.638042    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.660286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.660250    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"]
Oct 30 06:12:31.665134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.665099    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3abe42\"/\"openshift-service-ca.crt\""
Oct 30 06:12:31.665134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.665099    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3abe42\"/\"rate-test-model-3abe42-kserve-self-signed-certs\""
Oct 30 06:12:31.665395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.665208    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3abe42\"/\"kube-root-ca.crt\""
Oct 30 06:12:31.665556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.665530    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3abe42\"/\"default-dockercfg-z95zz\""
Oct 30 06:12:31.684054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.684024    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-model-cache\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.684166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.684073    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c436f8-7a93-4543-952e-35149302152f-tls-certs\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.684339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.684316    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-home\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.684483 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.684460    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-tmp-dir\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.684586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.684570    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsfc\" (UniqueName: \"kubernetes.io/projected/b3c436f8-7a93-4543-952e-35149302152f-kube-api-access-ffsfc\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.684700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.684689    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-dshm\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.719456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.719427    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:31.786332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.786295    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-dshm\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.786541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.786354    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-model-cache\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.786541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.786379    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c436f8-7a93-4543-952e-35149302152f-tls-certs\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.786541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.786455    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-home\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.786728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.786551    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-tmp-dir\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.786728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.786623    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsfc\" (UniqueName: \"kubernetes.io/projected/b3c436f8-7a93-4543-952e-35149302152f-kube-api-access-ffsfc\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.786862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.786810    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-model-cache\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.786918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.786872    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-home\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.787119 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.787067    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-tmp-dir\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.788870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.788850    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-dshm\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.789663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.789638    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c436f8-7a93-4543-952e-35149302152f-tls-certs\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.810679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.810649    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsfc\" (UniqueName: \"kubernetes.io/projected/b3c436f8-7a93-4543-952e-35149302152f-kube-api-access-ffsfc\") pod \"rate-test-model-3abe42-kserve-657db4db89-j6v65\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:31.962519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:31.962478    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:32.309109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:32.309079    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"]
Oct 30 06:12:32.310475 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:12:32.310444    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3c436f8_7a93_4543_952e_35149302152f.slice/crio-eef31063578ad2fb5ba0bcb53f8460aad63d4a4c6bd62a105520598b2e1101ae WatchSource:0}: Error finding container eef31063578ad2fb5ba0bcb53f8460aad63d4a4c6bd62a105520598b2e1101ae: Status 404 returned error can't find the container with id eef31063578ad2fb5ba0bcb53f8460aad63d4a4c6bd62a105520598b2e1101ae
Oct 30 06:12:32.377074 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:12:32.377044    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:12:32.879042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:32.879005    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:33.112799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:33.112774    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:12:33.196374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:33.196340    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65" event={"ID":"b3c436f8-7a93-4543-952e-35149302152f","Type":"ContainerStarted","Data":"dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7"}
Oct 30 06:12:33.196374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:33.196385    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65" event={"ID":"b3c436f8-7a93-4543-952e-35149302152f","Type":"ContainerStarted","Data":"eef31063578ad2fb5ba0bcb53f8460aad63d4a4c6bd62a105520598b2e1101ae"}
Oct 30 06:12:33.238322 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:33.238260    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65" podStartSLOduration=2.238243192 podStartE2EDuration="2.238243192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:33.235355285 +0000 UTC m=+2468.066392505" watchObservedRunningTime="2026-10-30 06:12:33.238243192 +0000 UTC m=+2468.069280410"
Oct 30 06:12:33.963630 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:33.963597    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:35.135572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:35.135534    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:36.020349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:36.020313    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:36.245039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:36.244990    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:37.083804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:37.083771    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:38.157082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:38.157056    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:12:38.913266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:38.913228    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:40.934160 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:40.934120    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:41.963650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:41.963616    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:41.976693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:41.976666    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:42.280362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:42.280279    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:12:42.902243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:42.902165    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:43.474807 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.474774    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"]
Oct 30 06:12:43.479649 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.479628    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.493506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.493486    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-1b4b7a-openshift-default-dockercfg-w77rm\""
Oct 30 06:12:43.508844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.508821    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"]
Oct 30 06:12:43.551372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551334    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.551531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551392    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.551531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551489    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jczc\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-kube-api-access-5jczc\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.551636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551599    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.551683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551668    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.551822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551806    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.551870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551837    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.551870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551855    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.551969 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.551961    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653168 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653133    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653279    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653327    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653380    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jczc\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-kube-api-access-5jczc\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653438    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653508    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653725    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653739    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653820    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653853    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.653950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.653876    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.654438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.654122    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.654438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.654391    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.654702 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.654677    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.656204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.656163    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.656336 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.656320    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.675115 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.675087    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.675251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.675224    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jczc\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-kube-api-access-5jczc\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:43.795594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:43.795507    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:44.124019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:44.123991    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"]
Oct 30 06:12:44.124597 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:12:44.124574    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f9fb67_bc26_41f0_bd53_1e63f3f9acdb.slice/crio-c25c49476ca2dbfda98545c0f5c23d0f705999c10a79178811617be9e2c6099e WatchSource:0}: Error finding container c25c49476ca2dbfda98545c0f5c23d0f705999c10a79178811617be9e2c6099e: Status 404 returned error can't find the container with id c25c49476ca2dbfda98545c0f5c23d0f705999c10a79178811617be9e2c6099e
Oct 30 06:12:44.127927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:44.127887    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:44.128030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:44.127977    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:44.128030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:44.128019    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:12:44.290592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:44.290551    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" event={"ID":"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb","Type":"ContainerStarted","Data":"8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf"}
Oct 30 06:12:44.290867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:44.290847    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" event={"ID":"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb","Type":"ContainerStarted","Data":"c25c49476ca2dbfda98545c0f5c23d0f705999c10a79178811617be9e2c6099e"}
Oct 30 06:12:44.749751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:44.749688    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:44.813216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:44.810834    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:45.798060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:45.798026    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 06:12:45.798499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:45.798084    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 06:12:46.554312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:46.554267    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:46.796430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:46.796399    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 06:12:46.796614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:46.796449    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 06:12:47.796086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:47.796050    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 06:12:47.796522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:47.796115    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 06:12:48.291329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:48.291294    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:48.796670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:48.796633    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 06:12:48.797082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:48.796686    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 06:12:49.389931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.389879    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" podStartSLOduration=6.389864697 podStartE2EDuration="6.389864697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:44.346554311 +0000 UTC m=+2479.177591532" watchObservedRunningTime="2026-10-30 06:12:49.389864697 +0000 UTC m=+2484.220901913"
Oct 30 06:12:49.391824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.391801    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"]
Oct 30 06:12:49.419485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.419451    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"]
Oct 30 06:12:49.547672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.547640    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:49.687033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.686916    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:49.796742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.796706    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 06:12:49.797261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.796788    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 06:12:49.825054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.825026    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:49.967218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:49.967097    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:50.870661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:50.870632    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:50.871221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:50.871199    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:50.872652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:50.872626    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:12:51.681329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:51.681293    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:53.604687 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:53.604652    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:54.393033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.392989    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh" podUID="1ccb90a7-36f7-47a9-aa7b-d2a280bba926" containerName="payload-processing" containerID="cri-o://f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb" gracePeriod=25
Oct 30 06:12:54.419837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.419786    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n" podUID="febdd55a-454d-472c-8ba7-fa5d99b48ea0" containerName="payload-pre-processing" containerID="cri-o://d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd" gracePeriod=25
Oct 30 06:12:54.725402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.725366    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:12:54.751117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.751094    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:12:54.868732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.868699    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febdd55a-454d-472c-8ba7-fa5d99b48ea0-kube-api-access-q5nm8\" (UniqueName: \"kubernetes.io/projected/febdd55a-454d-472c-8ba7-fa5d99b48ea0-kube-api-access-q5nm8\") pod \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\" (UID: \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\") "
Oct 30 06:12:54.868907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.868810    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/febdd55a-454d-472c-8ba7-fa5d99b48ea0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/febdd55a-454d-472c-8ba7-fa5d99b48ea0-plugins-config-volume\") pod \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\" (UID: \"febdd55a-454d-472c-8ba7-fa5d99b48ea0\") "
Oct 30 06:12:54.868907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.868838    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-plugins-config-volume\") pod \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\" (UID: \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\") "
Oct 30 06:12:54.868907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.868882    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-kube-api-access-g6xrm\" (UniqueName: \"kubernetes.io/projected/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-kube-api-access-g6xrm\") pod \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\" (UID: \"1ccb90a7-36f7-47a9-aa7b-d2a280bba926\") "
Oct 30 06:12:54.869266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.869238    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-plugins-config-volume" pod "1ccb90a7-36f7-47a9-aa7b-d2a280bba926" (UID: "1ccb90a7-36f7-47a9-aa7b-d2a280bba926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:12:54.869330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.869273    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febdd55a-454d-472c-8ba7-fa5d99b48ea0-plugins-config-volume" pod "febdd55a-454d-472c-8ba7-fa5d99b48ea0" (UID: "febdd55a-454d-472c-8ba7-fa5d99b48ea0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:12:54.869635 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.869617    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/febdd55a-454d-472c-8ba7-fa5d99b48ea0-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:54.869709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.869639    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:54.871099 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.871076    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febdd55a-454d-472c-8ba7-fa5d99b48ea0-kube-api-access-q5nm8" pod "febdd55a-454d-472c-8ba7-fa5d99b48ea0" (UID: "febdd55a-454d-472c-8ba7-fa5d99b48ea0"). InnerVolumeSpecName "kube-api-access-q5nm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:12:54.871166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.871141    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-kube-api-access-g6xrm" pod "1ccb90a7-36f7-47a9-aa7b-d2a280bba926" (UID: "1ccb90a7-36f7-47a9-aa7b-d2a280bba926"). InnerVolumeSpecName "kube-api-access-g6xrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:12:54.971091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.971017    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5nm8\" (UniqueName: \"kubernetes.io/projected/febdd55a-454d-472c-8ba7-fa5d99b48ea0-kube-api-access-q5nm8\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:54.971091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:54.971047    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6xrm\" (UniqueName: \"kubernetes.io/projected/1ccb90a7-36f7-47a9-aa7b-d2a280bba926-kube-api-access-g6xrm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:12:55.397733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.397698    5250 generic.go:356] "Generic (PLEG): container finished" podID="febdd55a-454d-472c-8ba7-fa5d99b48ea0" containerID="d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd" exitCode=0
Oct 30 06:12:55.397892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.397748    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"
Oct 30 06:12:55.397892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.397781    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n" event={"ID":"febdd55a-454d-472c-8ba7-fa5d99b48ea0","Type":"ContainerDied","Data":"d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd"}
Oct 30 06:12:55.397892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.397844    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n" event={"ID":"febdd55a-454d-472c-8ba7-fa5d99b48ea0","Type":"ContainerDied","Data":"3a7817caba4143f6989985f668e9b3a5b048e5e68e7b54796fa3b0bd59dd60f2"}
Oct 30 06:12:55.397892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.397873    5250 scope.go:122] "RemoveContainer" containerID="d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd"
Oct 30 06:12:55.400619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.400106    5250 generic.go:356] "Generic (PLEG): container finished" podID="1ccb90a7-36f7-47a9-aa7b-d2a280bba926" containerID="f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb" exitCode=0
Oct 30 06:12:55.400619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.400157    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"
Oct 30 06:12:55.400619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.400204    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh" event={"ID":"1ccb90a7-36f7-47a9-aa7b-d2a280bba926","Type":"ContainerDied","Data":"f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb"}
Oct 30 06:12:55.400619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.400246    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh" event={"ID":"1ccb90a7-36f7-47a9-aa7b-d2a280bba926","Type":"ContainerDied","Data":"3d099accef913952d205ab8f19db3dab6f835f7ab55785ad46a288fc1a22d761"}
Oct 30 06:12:55.426700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.421648    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:55.435456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.435436    5250 scope.go:122] "RemoveContainer" containerID="d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd"
Oct 30 06:12:55.435986 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:12:55.435950    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd\": container with ID starting with d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd not found: ID does not exist" containerID="d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd"
Oct 30 06:12:55.436104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.435995    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd"} err="failed to get container status \"d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd\": rpc error: code = NotFound desc = could not find container \"d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd\": container with ID starting with d9277be4bccb8982f1de0a5438c1158a18f63bc7b9cd8d776eefbaf62cb40efd not found: ID does not exist"
Oct 30 06:12:55.436104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.436019    5250 scope.go:122] "RemoveContainer" containerID="f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb"
Oct 30 06:12:55.451558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.450312    5250 scope.go:122] "RemoveContainer" containerID="f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb"
Oct 30 06:12:55.451558 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:12:55.450756    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb\": container with ID starting with f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb not found: ID does not exist" containerID="f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb"
Oct 30 06:12:55.451558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.450797    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb"} err="failed to get container status \"f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb\": rpc error: code = NotFound desc = could not find container \"f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb\": container with ID starting with f095ed8eccaa34505a304295e0a81c48543f7fb802ee1c3b9847ebd105acf0cb not found: ID does not exist"
Oct 30 06:12:55.462070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.462037    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"]
Oct 30 06:12:55.466698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.466668    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5dc5d6c5c9-4857n"]
Oct 30 06:12:55.498921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.498888    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"]
Oct 30 06:12:55.504736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:55.504712    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-6cffcf5f86-tgjsh"]
Oct 30 06:12:56.931455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:56.931408    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:57.284821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.284741    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ccb90a7-36f7-47a9-aa7b-d2a280bba926" path="/var/lib/kubelet/pods/1ccb90a7-36f7-47a9-aa7b-d2a280bba926/volumes"
Oct 30 06:12:57.285234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.285221    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febdd55a-454d-472c-8ba7-fa5d99b48ea0" path="/var/lib/kubelet/pods/febdd55a-454d-472c-8ba7-fa5d99b48ea0/volumes"
Oct 30 06:12:57.626921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.626845    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"]
Oct 30 06:12:57.627663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.627642    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1ccb90a7-36f7-47a9-aa7b-d2a280bba926" containerName="payload-processing"
Oct 30 06:12:57.627663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.627663    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ccb90a7-36f7-47a9-aa7b-d2a280bba926" containerName="payload-processing"
Oct 30 06:12:57.627827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.627675    5250 cpu_manager.go:414] "containerMap: removing container" podUID="febdd55a-454d-472c-8ba7-fa5d99b48ea0" containerName="payload-pre-processing"
Oct 30 06:12:57.627827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.627680    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febdd55a-454d-472c-8ba7-fa5d99b48ea0" containerName="payload-pre-processing"
Oct 30 06:12:57.627827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.627790    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ccb90a7-36f7-47a9-aa7b-d2a280bba926" containerName="payload-processing"
Oct 30 06:12:57.627827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.627801    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="febdd55a-454d-472c-8ba7-fa5d99b48ea0" containerName="payload-pre-processing"
Oct 30 06:12:57.633596 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.633570    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.656382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.656341    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"]
Oct 30 06:12:57.660542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.660512    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-2f1e4c\"/\"openshift-service-ca.crt\""
Oct 30 06:12:57.660721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.660631    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-2f1e4c\"/\"kube-root-ca.crt\""
Oct 30 06:12:57.668093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.668073    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-2f1e4c\"/\"sub-test-model-2f1e4c-kserve-self-signed-certs\""
Oct 30 06:12:57.668249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.668095    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-2f1e4c\"/\"default-dockercfg-bwkbq\""
Oct 30 06:12:57.710064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.710034    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-home\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.710222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.710077    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-dshm\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.710290 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.710225    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tls-certs\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.710364 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.710351    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tmp-dir\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.710481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.710467    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-model-cache\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.710543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.710492    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wrql\" (UniqueName: \"kubernetes.io/projected/b72959fa-4c54-4917-9f9d-27d00c97d4ce-kube-api-access-4wrql\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.720556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.720523    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:57.811512 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.811477    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-dshm\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.811708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.811576    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tls-certs\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.811797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.811731    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tmp-dir\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.811878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.811858    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-model-cache\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.811944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.811894    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrql\" (UniqueName: \"kubernetes.io/projected/b72959fa-4c54-4917-9f9d-27d00c97d4ce-kube-api-access-4wrql\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.812001 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.811988    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-home\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.812152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.812120    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tmp-dir\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.812532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.812386    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-home\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.812532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.812410    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-model-cache\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.814207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.814160    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-dshm\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.814734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.814713    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tls-certs\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.834601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.834573    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wrql\" (UniqueName: \"kubernetes.io/projected/b72959fa-4c54-4917-9f9d-27d00c97d4ce-kube-api-access-4wrql\") pod \"sub-test-model-2f1e4c-kserve-959978675-grb7n\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") " pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:57.963080 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:57.963041    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:12:58.316739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:58.316711    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"]
Oct 30 06:12:58.318295 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:12:58.318258    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb72959fa_4c54_4917_9f9d_27d00c97d4ce.slice/crio-6404e9e9278b6433a26e2889d5755c3ad00d590ab2337cf14862b57d2c55b9e7 WatchSource:0}: Error finding container 6404e9e9278b6433a26e2889d5755c3ad00d590ab2337cf14862b57d2c55b9e7: Status 404 returned error can't find the container with id 6404e9e9278b6433a26e2889d5755c3ad00d590ab2337cf14862b57d2c55b9e7
Oct 30 06:12:58.417413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:58.417382    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:58.451250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:58.449717    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n" event={"ID":"b72959fa-4c54-4917-9f9d-27d00c97d4ce","Type":"ContainerStarted","Data":"998a84dd7101a59e9448de1312a2ba77b00e2f843baead3381d4f92ae7f4530b"}
Oct 30 06:12:58.451250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:58.449771    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n" event={"ID":"b72959fa-4c54-4917-9f9d-27d00c97d4ce","Type":"ContainerStarted","Data":"6404e9e9278b6433a26e2889d5755c3ad00d590ab2337cf14862b57d2c55b9e7"}
Oct 30 06:12:58.493661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:58.493609    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n" podStartSLOduration=1.493595818 podStartE2EDuration="1.493595818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:58.488006433 +0000 UTC m=+2493.319043650" watchObservedRunningTime="2026-10-30 06:12:58.493595818 +0000 UTC m=+2493.324633053"
Oct 30 06:12:58.863369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:58.863331    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:12:58.963695 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:12:58.963667    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:13:00.504092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:00.504061    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:00.655067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:00.655035    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:02.281903 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:02.281872    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:02.375387 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:13:02.375352    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:13:03.070215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:03.069470    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"]
Oct 30 06:13:03.085203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:03.074409    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0de9ac-2nxzn"]
Oct 30 06:13:03.307448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:03.303129    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db95a07-1966-4eff-93c8-c88baec91b3f" path="/var/lib/kubelet/pods/4db95a07-1966-4eff-93c8-c88baec91b3f/volumes"
Oct 30 06:13:04.873619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:04.873586    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:07.111104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:07.111070    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:07.963701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:07.963666    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:13:07.976832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:07.976807    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:13:08.565299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.565269    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:13:08.850245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.850194    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"]
Oct 30 06:13:08.855926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.855901    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.869581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.869550    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ebe61e-serving-cert\""
Oct 30 06:13:08.874653 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.874626    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"]
Oct 30 06:13:08.895392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.895341    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.895651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.895515    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9vj7\" (UniqueName: \"kubernetes.io/projected/6ea4e14a-9d8b-4460-8bfb-21f904305667-kube-api-access-h9vj7\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.895752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.895712    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-metrics-certs\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.895824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.895784    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea4e14a-9d8b-4460-8bfb-21f904305667-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.995228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.995169    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:08.996652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.996627    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9vj7\" (UniqueName: \"kubernetes.io/projected/6ea4e14a-9d8b-4460-8bfb-21f904305667-kube-api-access-h9vj7\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.996757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.996744    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-metrics-certs\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.996802 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.996776    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea4e14a-9d8b-4460-8bfb-21f904305667-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.996862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.996851    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.997014 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:13:08.997002    5250 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-serving-cert: secret "maas-api-e2e-ipp-a-ebe61e-serving-cert" not found
Oct 30 06:13:08.997071 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:13:08.997063    5250 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls podName:6ea4e14a-9d8b-4460-8bfb-21f904305667 nodeName:}" failed. No retries permitted until 2026-10-30 06:13:09.497049017 +0000 UTC m=+2504.328086225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls") pod "maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm" (UID: "6ea4e14a-9d8b-4460-8bfb-21f904305667") : secret "maas-api-e2e-ipp-a-ebe61e-serving-cert" not found
Oct 30 06:13:08.997552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.997533    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea4e14a-9d8b-4460-8bfb-21f904305667-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:08.999530 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:08.999507    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-metrics-certs\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:09.019490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:09.019459    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9vj7\" (UniqueName: \"kubernetes.io/projected/6ea4e14a-9d8b-4460-8bfb-21f904305667-kube-api-access-h9vj7\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:09.141566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:09.141519    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:09.505460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:09.505359    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:09.508403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:09.508375    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:09.508729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:09.508687    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:09.771913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:09.771827    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:10.109456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:10.109368    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"]
Oct 30 06:13:10.114554 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:13:10.114520    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea4e14a_9d8b_4460_8bfb_21f904305667.slice/crio-6e197691a5b8d53cd1e65273b2ffe25638a541fea57bbe461e7644f5b5510dfc WatchSource:0}: Error finding container 6e197691a5b8d53cd1e65273b2ffe25638a541fea57bbe461e7644f5b5510dfc: Status 404 returned error can't find the container with id 6e197691a5b8d53cd1e65273b2ffe25638a541fea57bbe461e7644f5b5510dfc
Oct 30 06:13:10.116321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:10.116300    5250 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 06:13:10.573333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:10.573296    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm" event={"ID":"6ea4e14a-9d8b-4460-8bfb-21f904305667","Type":"ContainerStarted","Data":"f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e"}
Oct 30 06:13:10.573524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:10.573360    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm" event={"ID":"6ea4e14a-9d8b-4460-8bfb-21f904305667","Type":"ContainerStarted","Data":"6e197691a5b8d53cd1e65273b2ffe25638a541fea57bbe461e7644f5b5510dfc"}
Oct 30 06:13:10.573524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:10.573391    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:10.612786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:10.612726    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm" podStartSLOduration=2.265498795 podStartE2EDuration="2.612710162s" totalImagesPullingTime="347.211367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:10.612166551 +0000 UTC m=+2505.443203769" watchObservedRunningTime="2026-10-30 06:13:10.612710162 +0000 UTC m=+2505.443747379"
Oct 30 06:13:11.010712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:11.010653    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:12.919220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:12.919147    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:14.036000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:14.035961    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"]
Oct 30 06:13:14.045717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:14.045689    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a34f05-68zqd"]
Oct 30 06:13:14.544580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:14.544546    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:15.284795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:15.284763    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad85965-824b-4e18-8cb7-3df26685c0ff" path="/var/lib/kubelet/pods/8ad85965-824b-4e18-8cb7-3df26685c0ff/volumes"
Oct 30 06:13:16.116872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:16.116840    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:16.587338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:16.587307    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:13:18.349541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:18.349490    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:19.630560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.630529    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"]
Oct 30 06:13:19.636020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.635999    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.659255 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.659222    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-3abe42-openshift-default-dockercfg-6sv77\""
Oct 30 06:13:19.666516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.666486    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"]
Oct 30 06:13:19.756720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.756690    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.756887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.756751    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.756887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.756865    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.756967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.756916    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.757049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.757030    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.757095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.757084    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bd2dfe-d70f-4fd9-b75b-518282051732-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.757139 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.757128    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vdnf\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-kube-api-access-9vdnf\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.757219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.757204    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.757264 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.757241    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.858901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.858867    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.858949    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859006    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859082    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859117    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bd2dfe-d70f-4fd9-b75b-518282051732-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859234    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vdnf\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-kube-api-access-9vdnf\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859330    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859338    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859394    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859459    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859628    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859701    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859813 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859774    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.859898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.859881    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bd2dfe-d70f-4fd9-b75b-518282051732-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.861618 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.861595    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.861951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.861934    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.880624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.880565    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.880820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.880801    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vdnf\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-kube-api-access-9vdnf\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:19.951625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:19.951600    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:20.277524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.277497    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"]
Oct 30 06:13:20.278741 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:13:20.278712    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bd2dfe_d70f_4fd9_b75b_518282051732.slice/crio-c46b935e862f18a3d37a4fd0e20df778bee6164dd4c07b34a0d5a8c3974fb632 WatchSource:0}: Error finding container c46b935e862f18a3d37a4fd0e20df778bee6164dd4c07b34a0d5a8c3974fb632: Status 404 returned error can't find the container with id c46b935e862f18a3d37a4fd0e20df778bee6164dd4c07b34a0d5a8c3974fb632
Oct 30 06:13:20.281444 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.281409    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:20.281533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.281494    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:20.281589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.281539    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:20.334499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.334472    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:20.662243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.662171    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" event={"ID":"80bd2dfe-d70f-4fd9-b75b-518282051732","Type":"ContainerStarted","Data":"c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3"}
Oct 30 06:13:20.662773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.662275    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" event={"ID":"80bd2dfe-d70f-4fd9-b75b-518282051732","Type":"ContainerStarted","Data":"c46b935e862f18a3d37a4fd0e20df778bee6164dd4c07b34a0d5a8c3974fb632"}
Oct 30 06:13:20.703989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.703940    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" podStartSLOduration=1.7039252820000002 podStartE2EDuration="1.703925282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:20.703409773 +0000 UTC m=+2515.534446994" watchObservedRunningTime="2026-10-30 06:13:20.703925282 +0000 UTC m=+2515.534962498"
Oct 30 06:13:20.952134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:20.952052    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:21.953026 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:21.952992    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 06:13:21.953448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:21.953044    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 06:13:22.283722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:22.283596    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:22.952405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:22.952373    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 06:13:22.952604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:22.952427    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 06:13:23.952453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:23.952420    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 06:13:23.952855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:23.952472    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 06:13:24.141231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:24.141165    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:24.952547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:24.952505    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 06:13:24.953046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:24.952573    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 06:13:25.790616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:25.789383    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:25.952373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:25.952338    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 06:13:25.952528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:25.952381    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 06:13:26.952675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:26.952637    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 06:13:26.953300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:26.952700    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 06:13:27.473789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:27.473743    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:27.957752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:27.957720    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:27.958206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:27.958055    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:27.958791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:27.958772    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:13:28.124772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.124740    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"]
Oct 30 06:13:28.130015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.129985    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.149211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.148235    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-452a59\"/\"openshift-service-ca.crt\""
Oct 30 06:13:28.149211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.148520    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-452a59\"/\"rate-test-model-452a59-kserve-self-signed-certs\""
Oct 30 06:13:28.149211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.149156    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-452a59\"/\"kube-root-ca.crt\""
Oct 30 06:13:28.156414 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.156385    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-452a59\"/\"default-dockercfg-d2q44\""
Oct 30 06:13:28.163148 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.163106    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"]
Oct 30 06:13:28.195273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.195241    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tmp-dir\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.195408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.195303    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-model-cache\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.195473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.195452    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmm9\" (UniqueName: \"kubernetes.io/projected/99671d1e-c801-4908-a9c0-2f68fbea8ed3-kube-api-access-xrmm9\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.195566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.195554    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-dshm\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.195609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.195588    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-home\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.195658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.195627    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tls-certs\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.222751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.222676    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:28.267257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.267227    5250 scope.go:122] "RemoveContainer" containerID="ae31a2abab75847580fbf7c162a4d900007fa4777f259c9f00a4b8f153eb0c47"
Oct 30 06:13:28.281411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.281388    5250 scope.go:122] "RemoveContainer" containerID="2f66addbdbf83385ca0428b691b0013e97b947ef6647ff577316afc0789ad84e"
Oct 30 06:13:28.293785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.293760    5250 scope.go:122] "RemoveContainer" containerID="ffbffec34f985c665374147b4927cc643846f179466ee2cb6e10ea01160b6dd0"
Oct 30 06:13:28.296735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.296705    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-home\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.296990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.296945    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tls-certs\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.297094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.297039    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-home\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.297170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.297151    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tmp-dir\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.297303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.297285    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-model-cache\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.297430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.297409    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmm9\" (UniqueName: \"kubernetes.io/projected/99671d1e-c801-4908-a9c0-2f68fbea8ed3-kube-api-access-xrmm9\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.297526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.297508    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-dshm\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.297687 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.297664    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tmp-dir\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.297793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.297767    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-model-cache\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.300079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.300051    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-dshm\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.301426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.301401    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tls-certs\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.305452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.305429    5250 scope.go:122] "RemoveContainer" containerID="9052d8cfec4d85d77235d21ba80be0ab33121bee933126df70005ff17e10faf7"
Oct 30 06:13:28.320733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.320691    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmm9\" (UniqueName: \"kubernetes.io/projected/99671d1e-c801-4908-a9c0-2f68fbea8ed3-kube-api-access-xrmm9\") pod \"rate-test-model-452a59-kserve-758889db8f-hfwxd\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") " pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.472201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.462436    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:28.826462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:28.826432    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"]
Oct 30 06:13:28.827619 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:13:28.827591    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99671d1e_c801_4908_a9c0_2f68fbea8ed3.slice/crio-f180eb074e824aada40f82a1943a6af6f08a5b344c2c5a143713c55c92bada44 WatchSource:0}: Error finding container f180eb074e824aada40f82a1943a6af6f08a5b344c2c5a143713c55c92bada44: Status 404 returned error can't find the container with id f180eb074e824aada40f82a1943a6af6f08a5b344c2c5a143713c55c92bada44
Oct 30 06:13:29.501281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:29.501245    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:29.753606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:29.753504    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd" event={"ID":"99671d1e-c801-4908-a9c0-2f68fbea8ed3","Type":"ContainerStarted","Data":"113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659"}
Oct 30 06:13:29.753606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:29.753561    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd" event={"ID":"99671d1e-c801-4908-a9c0-2f68fbea8ed3","Type":"ContainerStarted","Data":"f180eb074e824aada40f82a1943a6af6f08a5b344c2c5a143713c55c92bada44"}
Oct 30 06:13:29.793080 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:29.793030    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd" podStartSLOduration=1.7930154040000001 podStartE2EDuration="1.793015404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:29.790760258 +0000 UTC m=+2524.621797475" watchObservedRunningTime="2026-10-30 06:13:29.793015404 +0000 UTC m=+2524.624052624"
Oct 30 06:13:30.462961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:30.462924    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:30.737887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:30.737764    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:30.891230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:30.889330    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:31.057277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.057047    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"]
Oct 30 06:13:31.063550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.063514    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.093243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.091273    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"]
Oct 30 06:13:31.093243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.091386    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-720e9c\"/\"kube-root-ca.crt\""
Oct 30 06:13:31.093243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.092028    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-720e9c\"/\"openshift-service-ca.crt\""
Oct 30 06:13:31.093243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.092095    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-720e9c\"/\"auth-test-model-720e9c-kserve-self-signed-certs\""
Oct 30 06:13:31.093243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.092259    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-720e9c\"/\"default-dockercfg-7c4ht\""
Oct 30 06:13:31.162881 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.162842    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:31.183595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.183565    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tls-certs\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.183772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.183638    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-model-cache\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.183772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.183738    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tmp-dir\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.183862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.183769    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-home\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.183910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.183886    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-dshm\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.184051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.184037    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrg7\" (UniqueName: \"kubernetes.io/projected/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-kube-api-access-jhrg7\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.285423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.285384    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-dshm\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.285635 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.285522    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrg7\" (UniqueName: \"kubernetes.io/projected/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-kube-api-access-jhrg7\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.285701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.285648    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tls-certs\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.285771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.285704    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-model-cache\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.285771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.285756    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tmp-dir\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.285883 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.285784    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-home\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.286367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.286335    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-model-cache\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.286499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.286391    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tmp-dir\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.286499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.286396    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-home\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.288048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.288021    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-dshm\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.288936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.288918    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tls-certs\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.310508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.310446    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrg7\" (UniqueName: \"kubernetes.io/projected/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-kube-api-access-jhrg7\") pod \"auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") " pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.426166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.417513    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:31.903693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.903663    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"]
Oct 30 06:13:31.905073 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:13:31.905046    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fa398e5_eadb_4eae_b34e_d3e36bf0d941.slice/crio-2da267b3d3ea42a31fa591663a4c67f39be56d57c012283352bdffc70177fdf2 WatchSource:0}: Error finding container 2da267b3d3ea42a31fa591663a4c67f39be56d57c012283352bdffc70177fdf2: Status 404 returned error can't find the container with id 2da267b3d3ea42a31fa591663a4c67f39be56d57c012283352bdffc70177fdf2
Oct 30 06:13:31.981936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:31.981912    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:32.375376 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:13:32.375287    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:13:32.779776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:32.779635    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw" event={"ID":"4fa398e5-eadb-4eae-b34e-d3e36bf0d941","Type":"ContainerStarted","Data":"14883aafdd0a14478b2ece2499e36d7858daf5bc53b1e315339f8c6864ef37a6"}
Oct 30 06:13:32.779776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:32.779688    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw" event={"ID":"4fa398e5-eadb-4eae-b34e-d3e36bf0d941","Type":"ContainerStarted","Data":"2da267b3d3ea42a31fa591663a4c67f39be56d57c012283352bdffc70177fdf2"}
Oct 30 06:13:32.818468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:32.818411    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw" podStartSLOduration=1.818392021 podStartE2EDuration="1.818392021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:32.816792777 +0000 UTC m=+2527.647830006" watchObservedRunningTime="2026-10-30 06:13:32.818392021 +0000 UTC m=+2527.649429236"
Oct 30 06:13:33.031889 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:33.031811    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"]
Oct 30 06:13:33.035539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:33.035513    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df18371e-8vptp"]
Oct 30 06:13:33.282915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:33.282834    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff983fd-753b-464a-8655-f5c9b84665cc" path="/var/lib/kubelet/pods/0ff983fd-753b-464a-8655-f5c9b84665cc/volumes"
Oct 30 06:13:33.417964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:33.417928    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:33.693121 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:33.693072    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:35.670237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:35.670204    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:36.133114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:36.133079    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:36.345735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:36.345701    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:37.316209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:37.315863    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:38.463733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:38.463698    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:38.476784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:38.476759    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:38.867499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:38.867412    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:13:39.090678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:39.090644    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:40.983205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:40.983030    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:41.419207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:41.418581    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:41.433044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:41.432745    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:41.945640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:41.945610    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:13:42.207260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.206230    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"]
Oct 30 06:13:42.220350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.216448    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.230594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.230563    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"]
Oct 30 06:13:42.238133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.237867    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:13:42.238133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.237979    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:13:42.250731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.250706    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2fnk\" (UniqueName: \"kubernetes.io/projected/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-kube-api-access-w2fnk\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.251028 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.251014    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.251157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.251147    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.352557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.352524    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.352911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.352884    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.353118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.353096    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2fnk\" (UniqueName: \"kubernetes.io/projected/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-kube-api-access-w2fnk\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.353425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.353406    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.359602 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.359578    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.385218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.376848    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2fnk\" (UniqueName: \"kubernetes.io/projected/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-kube-api-access-w2fnk\") pod \"maas-api-revoke-sub-3661c28d-vng59\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.540885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.540804    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:42.667402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:42.664739    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:43.151214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:43.143898    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"]
Oct 30 06:13:43.151214 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:13:43.148538    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cccff8_cf87_497e_aeee_ca0bf0ff44d3.slice/crio-94f83213f9ee2360225042681549a542738c65b7bc6090fb50998bee5589ea2e WatchSource:0}: Error finding container 94f83213f9ee2360225042681549a542738c65b7bc6090fb50998bee5589ea2e: Status 404 returned error can't find the container with id 94f83213f9ee2360225042681549a542738c65b7bc6090fb50998bee5589ea2e
Oct 30 06:13:44.029792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:44.029710    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59" event={"ID":"71cccff8-cf87-497e-aeee-ca0bf0ff44d3","Type":"ContainerStarted","Data":"7b32592f3d482c8dbd9435af55890b54ed67d9ca136207e871c89c323088a1bb"}
Oct 30 06:13:44.029792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:44.029774    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59" event={"ID":"71cccff8-cf87-497e-aeee-ca0bf0ff44d3","Type":"ContainerStarted","Data":"94f83213f9ee2360225042681549a542738c65b7bc6090fb50998bee5589ea2e"}
Oct 30 06:13:44.095204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:44.094577    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59" podStartSLOduration=2.094557992 podStartE2EDuration="2.094557992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:44.080479312 +0000 UTC m=+2538.911516531" watchObservedRunningTime="2026-10-30 06:13:44.094557992 +0000 UTC m=+2538.925595210"
Oct 30 06:13:44.312318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:44.311984    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:45.039067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:45.039032    5250 generic.go:356] "Generic (PLEG): container finished" podID="71cccff8-cf87-497e-aeee-ca0bf0ff44d3" containerID="7b32592f3d482c8dbd9435af55890b54ed67d9ca136207e871c89c323088a1bb" exitCode=0
Oct 30 06:13:45.039556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:45.039102    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59" event={"ID":"71cccff8-cf87-497e-aeee-ca0bf0ff44d3","Type":"ContainerDied","Data":"7b32592f3d482c8dbd9435af55890b54ed67d9ca136207e871c89c323088a1bb"}
Oct 30 06:13:45.982028 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:45.981678    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:46.265096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.265027    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:46.369590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.369135    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-kube-api-access-w2fnk\" (UniqueName: \"kubernetes.io/projected/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-kube-api-access-w2fnk\") pod \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") "
Oct 30 06:13:46.369590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.369266    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-xks-ca\") pod \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") "
Oct 30 06:13:46.369590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.369518    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-openshift-ca\") pod \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\" (UID: \"71cccff8-cf87-497e-aeee-ca0bf0ff44d3\") "
Oct 30 06:13:46.371813 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.371779    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-openshift-ca" pod "71cccff8-cf87-497e-aeee-ca0bf0ff44d3" (UID: "71cccff8-cf87-497e-aeee-ca0bf0ff44d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:13:46.373053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.373013    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-kube-api-access-w2fnk" pod "71cccff8-cf87-497e-aeee-ca0bf0ff44d3" (UID: "71cccff8-cf87-497e-aeee-ca0bf0ff44d3"). InnerVolumeSpecName "kube-api-access-w2fnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:13:46.373201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.373081    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-xks-ca" pod "71cccff8-cf87-497e-aeee-ca0bf0ff44d3" (UID: "71cccff8-cf87-497e-aeee-ca0bf0ff44d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:13:46.472150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.472095    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:13:46.472150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.472123    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2fnk\" (UniqueName: \"kubernetes.io/projected/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-kube-api-access-w2fnk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:13:46.472150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:46.472142    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71cccff8-cf87-497e-aeee-ca0bf0ff44d3-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:13:47.064070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:47.064034    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59" event={"ID":"71cccff8-cf87-497e-aeee-ca0bf0ff44d3","Type":"ContainerDied","Data":"94f83213f9ee2360225042681549a542738c65b7bc6090fb50998bee5589ea2e"}
Oct 30 06:13:47.064316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:47.064096    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f83213f9ee2360225042681549a542738c65b7bc6090fb50998bee5589ea2e"
Oct 30 06:13:47.064316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:47.064166    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"
Oct 30 06:13:47.487630 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:47.487586    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:47.688154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:47.688069    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:47.951087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:47.949500    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:48.087215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.081410    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:13:48.087215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.081479    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:13:48.241223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.240967    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:48.603300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.601609    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"]
Oct 30 06:13:48.603300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.602819    5250 cpu_manager.go:414] "containerMap: removing container" podUID="71cccff8-cf87-497e-aeee-ca0bf0ff44d3" containerName="revoke-keys"
Oct 30 06:13:48.603300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.602840    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cccff8-cf87-497e-aeee-ca0bf0ff44d3" containerName="revoke-keys"
Oct 30 06:13:48.603916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.603105    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cccff8-cf87-497e-aeee-ca0bf0ff44d3" containerName="revoke-keys"
Oct 30 06:13:48.608508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.608481    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.623372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.623343    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-95498c\"/\"openshift-service-ca.crt\""
Oct 30 06:13:48.623569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.623475    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-95498c\"/\"default-dockercfg-9zbmq\""
Oct 30 06:13:48.624282 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.624232    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-95498c\"/\"auth-test-model-95498c-kserve-self-signed-certs\""
Oct 30 06:13:48.624463 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.624448    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-95498c\"/\"kube-root-ca.crt\""
Oct 30 06:13:48.631285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.631262    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"]
Oct 30 06:13:48.678714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.678684    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:48.734346 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.734312    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-home\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.734526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.734469    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tmp-dir\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.734586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.734550    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-dshm\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.734772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.734747    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-model-cache\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.734841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.734822    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tls-certs\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.734926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.734909    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85ns\" (UniqueName: \"kubernetes.io/projected/b60419d0-842b-4dd4-bdcd-df58427cf9d7-kube-api-access-x85ns\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.836471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.836413    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-home\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.836692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.836510    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tmp-dir\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.836692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.836575    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-dshm\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.836844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.836712    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-model-cache\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.836844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.836772    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tls-certs\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.836965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.836847    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85ns\" (UniqueName: \"kubernetes.io/projected/b60419d0-842b-4dd4-bdcd-df58427cf9d7-kube-api-access-x85ns\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.837615 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.837586    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-model-cache\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.837900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.837881    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-home\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.838228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.838168    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tmp-dir\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.847809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.847453    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-dshm\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.847809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.847760    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tls-certs\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.919965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.913905    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85ns\" (UniqueName: \"kubernetes.io/projected/b60419d0-842b-4dd4-bdcd-df58427cf9d7-kube-api-access-x85ns\") pod \"auth-test-model-95498c-kserve-d46cbc545-tfx9c\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") " pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:48.960012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:48.959808    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:49.480317 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:49.479061    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:49.554226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:49.554199    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"]
Oct 30 06:13:49.556096 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:13:49.556069    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb60419d0_842b_4dd4_bdcd_df58427cf9d7.slice/crio-a38ebc595d737f27cfadcfc59ce6f8f605cf945a02355543dc1cf5798510498e WatchSource:0}: Error finding container a38ebc595d737f27cfadcfc59ce6f8f605cf945a02355543dc1cf5798510498e: Status 404 returned error can't find the container with id a38ebc595d737f27cfadcfc59ce6f8f605cf945a02355543dc1cf5798510498e
Oct 30 06:13:50.308899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.308868    5250 generic.go:356] "Generic (PLEG): container finished" podID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerID="e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e" exitCode=0
Oct 30 06:13:50.310372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.308949    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" event={"ID":"2f40cc9b-5c8e-414e-8b57-223c8b18b40f","Type":"ContainerDied","Data":"e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e"}
Oct 30 06:13:50.310372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.309535    5250 scope.go:122] "RemoveContainer" containerID="e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e"
Oct 30 06:13:50.311154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.311123    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c" event={"ID":"b60419d0-842b-4dd4-bdcd-df58427cf9d7","Type":"ContainerStarted","Data":"d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955"}
Oct 30 06:13:50.311310 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.311167    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c" event={"ID":"b60419d0-842b-4dd4-bdcd-df58427cf9d7","Type":"ContainerStarted","Data":"a38ebc595d737f27cfadcfc59ce6f8f605cf945a02355543dc1cf5798510498e"}
Oct 30 06:13:50.312131 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.312095    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:50.312313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.312210    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:50.312313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.312263    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:50.400030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.399954    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c" podStartSLOduration=2.399938488 podStartE2EDuration="2.399938488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:50.397349715 +0000 UTC m=+2545.228386932" watchObservedRunningTime="2026-10-30 06:13:50.399938488 +0000 UTC m=+2545.230975704"
Oct 30 06:13:50.960756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:50.960721    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:51.002091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.002051    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:51.120324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.120282    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"]
Oct 30 06:13:51.299250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.299086    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:51.320174 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.320145    5250 generic.go:356] "Generic (PLEG): container finished" podID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerID="f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b" exitCode=0
Oct 30 06:13:51.320633 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.320210    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" event={"ID":"966f242f-1a7a-4a18-93ae-66d06f77f88f","Type":"ContainerDied","Data":"f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b"}
Oct 30 06:13:51.320709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.320693    5250 scope.go:122] "RemoveContainer" containerID="f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b"
Oct 30 06:13:51.322853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.322824    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:51.322990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.322872    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" event={"ID":"2f40cc9b-5c8e-414e-8b57-223c8b18b40f","Type":"ContainerStarted","Data":"049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e"}
Oct 30 06:13:51.322990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.322910    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:51.322990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.322957    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:13:51.440222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.440172    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:51.636518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.636485    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:51.794202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.789019    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:51.997714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:51.997631    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:52.150506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.150475    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:52.296299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.296207    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:13:52.297827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.297801    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 06:13:52.297957 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.297852    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 06:13:52.334528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.334498    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" event={"ID":"966f242f-1a7a-4a18-93ae-66d06f77f88f","Type":"ContainerStarted","Data":"8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c"}
Oct 30 06:13:52.616370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.616280    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"]
Oct 30 06:13:52.620331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.620310    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.633571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.633547    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:13:52.633834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.633818    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:13:52.634797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.634771    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"]
Oct 30 06:13:52.722944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.722913    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.723254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.723237    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctc95\" (UniqueName: \"kubernetes.io/projected/1cf0a329-84b3-476f-9ca4-d064097e9814-kube-api-access-ctc95\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.723404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.723393    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.803976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.803681    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:13:52.807688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.806407    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 06:13:52.807688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.806479    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 06:13:52.825120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.825089    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.825326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.825214    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctc95\" (UniqueName: \"kubernetes.io/projected/1cf0a329-84b3-476f-9ca4-d064097e9814-kube-api-access-ctc95\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.825326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.825251    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.825799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.825780    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.827910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.827891    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.846944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.846918    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctc95\" (UniqueName: \"kubernetes.io/projected/1cf0a329-84b3-476f-9ca4-d064097e9814-kube-api-access-ctc95\") pod \"maas-api-revoke-sub-f07b940a-vwng7\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:52.933632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:52.933603    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:53.309253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.308373    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 06:13:53.309253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.308437    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 06:13:53.314400 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:13:53.314343    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf0a329_84b3_476f_9ca4_d064097e9814.slice/crio-2fdbe66111fdf2a53d51286f6d254de81f3b6ebb3db3d62d0658c1a42f24b357 WatchSource:0}: Error finding container 2fdbe66111fdf2a53d51286f6d254de81f3b6ebb3db3d62d0658c1a42f24b357: Status 404 returned error can't find the container with id 2fdbe66111fdf2a53d51286f6d254de81f3b6ebb3db3d62d0658c1a42f24b357
Oct 30 06:13:53.317492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.317470    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"]
Oct 30 06:13:53.345843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.345814    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7" event={"ID":"1cf0a329-84b3-476f-9ca4-d064097e9814","Type":"ContainerStarted","Data":"2fdbe66111fdf2a53d51286f6d254de81f3b6ebb3db3d62d0658c1a42f24b357"}
Oct 30 06:13:53.534379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.534333    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:53.623931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.623904    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"]
Oct 30 06:13:53.628412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.628391    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:53.640387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.640354    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x92w\" (UniqueName: \"kubernetes.io/projected/e6f19865-9040-40c3-aa79-93b1505c0224-kube-api-access-4x92w\") pod \"payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx\" (UID: \"e6f19865-9040-40c3-aa79-93b1505c0224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:53.640605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.640578    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f19865-9040-40c3-aa79-93b1505c0224-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx\" (UID: \"e6f19865-9040-40c3-aa79-93b1505c0224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:53.642158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.642127    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-370dfa\""
Oct 30 06:13:53.646410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.646381    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"]
Oct 30 06:13:53.716901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.716760    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:53.717684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.717653    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-370dfa-dockercfg-7dw2b\""
Oct 30 06:13:53.806817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.803686    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x92w\" (UniqueName: \"kubernetes.io/projected/e6f19865-9040-40c3-aa79-93b1505c0224-kube-api-access-4x92w\") pod \"payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx\" (UID: \"e6f19865-9040-40c3-aa79-93b1505c0224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:53.806817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.803856    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f19865-9040-40c3-aa79-93b1505c0224-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx\" (UID: \"e6f19865-9040-40c3-aa79-93b1505c0224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:53.806817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.805061    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f19865-9040-40c3-aa79-93b1505c0224-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx\" (UID: \"e6f19865-9040-40c3-aa79-93b1505c0224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:53.876254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.874005    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 06:13:53.876254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:53.874059    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 06:13:54.047137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.047092    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:54.079201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.076953    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x92w\" (UniqueName: \"kubernetes.io/projected/e6f19865-9040-40c3-aa79-93b1505c0224-kube-api-access-4x92w\") pod \"payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx\" (UID: \"e6f19865-9040-40c3-aa79-93b1505c0224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:54.125447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.124367    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:13:54.242820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.242788    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:54.313020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.312985    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 06:13:54.313779 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.313745    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 06:13:54.377835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.376651    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7" event={"ID":"1cf0a329-84b3-476f-9ca4-d064097e9814","Type":"ContainerStarted","Data":"bbe010621ceed714da992f50df024459e008e6155989f12af45ff101eb7d747a"}
Oct 30 06:13:54.419430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.419374    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7" podStartSLOduration=2.419359893 podStartE2EDuration="2.419359893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:54.41488317 +0000 UTC m=+2549.245920388" watchObservedRunningTime="2026-10-30 06:13:54.419359893 +0000 UTC m=+2549.250397111"
Oct 30 06:13:54.714222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.714166    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"]
Oct 30 06:13:54.715209 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:13:54.715162    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f19865_9040_40c3_aa79_93b1505c0224.slice/crio-ddd721110274f265205d0c9623775121bddad0f8a511e9f62f298fecb4beb80a WatchSource:0}: Error finding container ddd721110274f265205d0c9623775121bddad0f8a511e9f62f298fecb4beb80a: Status 404 returned error can't find the container with id ddd721110274f265205d0c9623775121bddad0f8a511e9f62f298fecb4beb80a
Oct 30 06:13:54.789768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.789310    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 06:13:54.789768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:54.789374    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 06:13:55.296753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.296716    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 06:13:55.297043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.297016    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 06:13:55.414943 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.414912    5250 generic.go:356] "Generic (PLEG): container finished" podID="1cf0a329-84b3-476f-9ca4-d064097e9814" containerID="bbe010621ceed714da992f50df024459e008e6155989f12af45ff101eb7d747a" exitCode=0
Oct 30 06:13:55.415448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.415157    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7" event={"ID":"1cf0a329-84b3-476f-9ca4-d064097e9814","Type":"ContainerDied","Data":"bbe010621ceed714da992f50df024459e008e6155989f12af45ff101eb7d747a"}
Oct 30 06:13:55.416991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.416968    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx" event={"ID":"e6f19865-9040-40c3-aa79-93b1505c0224","Type":"ContainerStarted","Data":"7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8"}
Oct 30 06:13:55.417948 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.417927    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:13:55.418139 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.418121    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx" event={"ID":"e6f19865-9040-40c3-aa79-93b1505c0224","Type":"ContainerStarted","Data":"ddd721110274f265205d0c9623775121bddad0f8a511e9f62f298fecb4beb80a"}
Oct 30 06:13:55.513880 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.513816    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx" podStartSLOduration=2.513796815 podStartE2EDuration="2.513796815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:55.505548753 +0000 UTC m=+2550.336585973" watchObservedRunningTime="2026-10-30 06:13:55.513796815 +0000 UTC m=+2550.344834034"
Oct 30 06:13:55.594129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.594001    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:55.777216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.776618    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 06:13:55.777216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:55.776686    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 06:13:56.121012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.120949    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc" podUID="d350f3ef-5d12-4021-9fa5-88e397430ecf" containerName="payload-processing" containerID="cri-o://70a9ffc79ef512015e06912dbd7229394bd5316807b18ce786776a43030f67cf" gracePeriod=25
Oct 30 06:13:56.311207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.308622    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 06:13:56.311207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.308688    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 06:13:56.430365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.430330    5250 generic.go:356] "Generic (PLEG): container finished" podID="d350f3ef-5d12-4021-9fa5-88e397430ecf" containerID="70a9ffc79ef512015e06912dbd7229394bd5316807b18ce786776a43030f67cf" exitCode=0
Oct 30 06:13:56.430836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.430811    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc" event={"ID":"d350f3ef-5d12-4021-9fa5-88e397430ecf","Type":"ContainerDied","Data":"70a9ffc79ef512015e06912dbd7229394bd5316807b18ce786776a43030f67cf"}
Oct 30 06:13:56.638514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.638432    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:13:56.734497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.734363    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d350f3ef-5d12-4021-9fa5-88e397430ecf-kube-api-access-gd2v8\" (UniqueName: \"kubernetes.io/projected/d350f3ef-5d12-4021-9fa5-88e397430ecf-kube-api-access-gd2v8\") pod \"d350f3ef-5d12-4021-9fa5-88e397430ecf\" (UID: \"d350f3ef-5d12-4021-9fa5-88e397430ecf\") "
Oct 30 06:13:56.734497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.734436    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d350f3ef-5d12-4021-9fa5-88e397430ecf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d350f3ef-5d12-4021-9fa5-88e397430ecf-plugins-config-volume\") pod \"d350f3ef-5d12-4021-9fa5-88e397430ecf\" (UID: \"d350f3ef-5d12-4021-9fa5-88e397430ecf\") "
Oct 30 06:13:56.735171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.734938    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d350f3ef-5d12-4021-9fa5-88e397430ecf-plugins-config-volume" pod "d350f3ef-5d12-4021-9fa5-88e397430ecf" (UID: "d350f3ef-5d12-4021-9fa5-88e397430ecf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:13:56.735616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.735350    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d350f3ef-5d12-4021-9fa5-88e397430ecf-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:13:56.740538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.740512    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d350f3ef-5d12-4021-9fa5-88e397430ecf-kube-api-access-gd2v8" pod "d350f3ef-5d12-4021-9fa5-88e397430ecf" (UID: "d350f3ef-5d12-4021-9fa5-88e397430ecf"). InnerVolumeSpecName "kube-api-access-gd2v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:13:56.799312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.788354    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 06:13:56.799312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.788420    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 06:13:56.837047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.837011    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2v8\" (UniqueName: \"kubernetes.io/projected/d350f3ef-5d12-4021-9fa5-88e397430ecf-kube-api-access-gd2v8\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:13:56.853852 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.853828    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:56.948218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.946805    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-openshift-ca\") pod \"1cf0a329-84b3-476f-9ca4-d064097e9814\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") "
Oct 30 06:13:56.948218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.947021    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-xks-ca\") pod \"1cf0a329-84b3-476f-9ca4-d064097e9814\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") "
Oct 30 06:13:56.948218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.947065    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf0a329-84b3-476f-9ca4-d064097e9814-kube-api-access-ctc95\" (UniqueName: \"kubernetes.io/projected/1cf0a329-84b3-476f-9ca4-d064097e9814-kube-api-access-ctc95\") pod \"1cf0a329-84b3-476f-9ca4-d064097e9814\" (UID: \"1cf0a329-84b3-476f-9ca4-d064097e9814\") "
Oct 30 06:13:56.952212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:56.949535    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-openshift-ca" pod "1cf0a329-84b3-476f-9ca4-d064097e9814" (UID: "1cf0a329-84b3-476f-9ca4-d064097e9814"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:13:57.015256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.008765    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf0a329-84b3-476f-9ca4-d064097e9814-kube-api-access-ctc95" pod "1cf0a329-84b3-476f-9ca4-d064097e9814" (UID: "1cf0a329-84b3-476f-9ca4-d064097e9814"). InnerVolumeSpecName "kube-api-access-ctc95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:13:57.015256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.010327    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-xks-ca" pod "1cf0a329-84b3-476f-9ca4-d064097e9814" (UID: "1cf0a329-84b3-476f-9ca4-d064097e9814"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:13:57.068201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.067825    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:13:57.068201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.067881    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf0a329-84b3-476f-9ca4-d064097e9814-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:13:57.068201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.067895    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctc95\" (UniqueName: \"kubernetes.io/projected/1cf0a329-84b3-476f-9ca4-d064097e9814-kube-api-access-ctc95\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:13:57.407763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.407727    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:57.440444 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.440414    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"
Oct 30 06:13:57.440941 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.440455    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc" event={"ID":"d350f3ef-5d12-4021-9fa5-88e397430ecf","Type":"ContainerDied","Data":"508105d5fa4984291ddcd0154a7aba324b62581edd4fa4e8f15ea7dd6939d837"}
Oct 30 06:13:57.440941 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.440534    5250 scope.go:122] "RemoveContainer" containerID="70a9ffc79ef512015e06912dbd7229394bd5316807b18ce786776a43030f67cf"
Oct 30 06:13:57.443015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.442996    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"
Oct 30 06:13:57.443812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.443785    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7" event={"ID":"1cf0a329-84b3-476f-9ca4-d064097e9814","Type":"ContainerDied","Data":"2fdbe66111fdf2a53d51286f6d254de81f3b6ebb3db3d62d0658c1a42f24b357"}
Oct 30 06:13:57.443914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.443835    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fdbe66111fdf2a53d51286f6d254de81f3b6ebb3db3d62d0658c1a42f24b357"
Oct 30 06:13:57.571258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.567234    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"]
Oct 30 06:13:57.597320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.597279    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-5b7f8f695b-jx7kc"]
Oct 30 06:13:57.599492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.599466    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:13:57.601559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.601534    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:13:57.781958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.781338    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 06:13:57.781958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:57.781409    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 06:13:58.778331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:58.778301    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:13:58.778824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:58.778804    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:13:58.779677 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:58.779660    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:13:58.960244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:58.960210    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:58.973212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:58.973172    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:13:59.207393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:59.207355    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:13:59.284218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:59.284171    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d350f3ef-5d12-4021-9fa5-88e397430ecf" path="/var/lib/kubelet/pods/d350f3ef-5d12-4021-9fa5-88e397430ecf/volumes"
Oct 30 06:13:59.474297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:13:59.474212    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:14:00.977934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:00.977905    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:01.915966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:01.915935    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"]
Oct 30 06:14:02.076979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:02.076930    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:02.224611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:02.224516    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:02.375591 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:14:02.375556    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:14:02.470729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:02.470695    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:02.693079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:02.693038    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:04.410897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:04.410857    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:05.996801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:05.996676    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:06.916454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:06.916411    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf" podUID="017b8f8e-f88d-48da-b101-7bd8721647aa" containerName="payload-processing" containerID="cri-o://604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee" gracePeriod=25
Oct 30 06:14:07.224902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.224877    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:14:07.258301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.258269    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017b8f8e-f88d-48da-b101-7bd8721647aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b8f8e-f88d-48da-b101-7bd8721647aa-plugins-config-volume\") pod \"017b8f8e-f88d-48da-b101-7bd8721647aa\" (UID: \"017b8f8e-f88d-48da-b101-7bd8721647aa\") "
Oct 30 06:14:07.258473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.258314    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017b8f8e-f88d-48da-b101-7bd8721647aa-kube-api-access-c95pb\" (UniqueName: \"kubernetes.io/projected/017b8f8e-f88d-48da-b101-7bd8721647aa-kube-api-access-c95pb\") pod \"017b8f8e-f88d-48da-b101-7bd8721647aa\" (UID: \"017b8f8e-f88d-48da-b101-7bd8721647aa\") "
Oct 30 06:14:07.258652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.258632    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017b8f8e-f88d-48da-b101-7bd8721647aa-plugins-config-volume" pod "017b8f8e-f88d-48da-b101-7bd8721647aa" (UID: "017b8f8e-f88d-48da-b101-7bd8721647aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:14:07.259005 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.258982    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017b8f8e-f88d-48da-b101-7bd8721647aa-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:07.260689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.260664    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017b8f8e-f88d-48da-b101-7bd8721647aa-kube-api-access-c95pb" pod "017b8f8e-f88d-48da-b101-7bd8721647aa" (UID: "017b8f8e-f88d-48da-b101-7bd8721647aa"). InnerVolumeSpecName "kube-api-access-c95pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:14:07.360848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.360814    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c95pb\" (UniqueName: \"kubernetes.io/projected/017b8f8e-f88d-48da-b101-7bd8721647aa-kube-api-access-c95pb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:07.446946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.446869    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:14:07.512981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.512937    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:07.534026 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.533995    5250 generic.go:356] "Generic (PLEG): container finished" podID="017b8f8e-f88d-48da-b101-7bd8721647aa" containerID="604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee" exitCode=0
Oct 30 06:14:07.534304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.534066    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"
Oct 30 06:14:07.534304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.534085    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf" event={"ID":"017b8f8e-f88d-48da-b101-7bd8721647aa","Type":"ContainerDied","Data":"604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee"}
Oct 30 06:14:07.534304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.534168    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf" event={"ID":"017b8f8e-f88d-48da-b101-7bd8721647aa","Type":"ContainerDied","Data":"4be5e97338f3227078c2328d8b0d71b80da75ced659d4e6504bea8b2b12a6a3a"}
Oct 30 06:14:07.534304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.534215    5250 scope.go:122] "RemoveContainer" containerID="604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee"
Oct 30 06:14:07.546657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.546637    5250 scope.go:122] "RemoveContainer" containerID="604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee"
Oct 30 06:14:07.547002 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:14:07.546980    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee\": container with ID starting with 604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee not found: ID does not exist" containerID="604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee"
Oct 30 06:14:07.547074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.547012    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee"} err="failed to get container status \"604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee\": rpc error: code = NotFound desc = could not find container \"604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee\": container with ID starting with 604449a6fd92f01a8dff071435a248723095adefa672538e601801cfc30098ee not found: ID does not exist"
Oct 30 06:14:07.584440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.584407    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"]
Oct 30 06:14:07.588834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:07.588807    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-69b5b96bd4-vjrwf"]
Oct 30 06:14:08.351675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:08.351638    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:09.179942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:09.179895    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:09.284919 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:09.284884    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017b8f8e-f88d-48da-b101-7bd8721647aa" path="/var/lib/kubelet/pods/017b8f8e-f88d-48da-b101-7bd8721647aa/volumes"
Oct 30 06:14:10.709693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:10.709651    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:11.053849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:11.053757    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:12.338959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:12.338905    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:14.204660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:14.204625    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:16.380227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:16.380135    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:17.995429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:17.995386    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:19.787872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:19.787813    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:21.370417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:21.370384    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:22.964763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:22.964731    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:24.515717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:24.515681    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:26.186114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:26.186072    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:28.439065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:28.439021    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:28.461293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:28.461115    5250 scope.go:122] "RemoveContainer" containerID="b69ee1174d0951743560f835eba881b6ab6549711dc3adeb8813c11e98b824fa"
Oct 30 06:14:28.481073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:28.480935    5250 scope.go:122] "RemoveContainer" containerID="aca082e8d6aa4f05ac5efe585b21b390533612545a76dd61db7b435d406d3031"
Oct 30 06:14:28.514414 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:28.514383    5250 scope.go:122] "RemoveContainer" containerID="4ceb7b93146b4a0038974240990584686f4b46cba4beacf7df63d1fc29b41661"
Oct 30 06:14:28.529357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:28.529329    5250 scope.go:122] "RemoveContainer" containerID="837cbf03e977b5a1ee6ca845c7efc0451648ee4748166375daee5c9bac1a2f34"
Oct 30 06:14:30.295820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:30.295780    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:32.037522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:32.037487    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:32.376300 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:14:32.376172    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:14:33.209506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:33.209477    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:33.387245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:33.387210    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:34.127441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.127407    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"]
Oct 30 06:14:34.128150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128134    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1cf0a329-84b3-476f-9ca4-d064097e9814" containerName="revoke-keys"
Oct 30 06:14:34.128226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128152    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf0a329-84b3-476f-9ca4-d064097e9814" containerName="revoke-keys"
Oct 30 06:14:34.128226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128192    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d350f3ef-5d12-4021-9fa5-88e397430ecf" containerName="payload-processing"
Oct 30 06:14:34.128226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128197    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d350f3ef-5d12-4021-9fa5-88e397430ecf" containerName="payload-processing"
Oct 30 06:14:34.128226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128221    5250 cpu_manager.go:414] "containerMap: removing container" podUID="017b8f8e-f88d-48da-b101-7bd8721647aa" containerName="payload-processing"
Oct 30 06:14:34.128226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128226    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017b8f8e-f88d-48da-b101-7bd8721647aa" containerName="payload-processing"
Oct 30 06:14:34.128435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128332    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf0a329-84b3-476f-9ca4-d064097e9814" containerName="revoke-keys"
Oct 30 06:14:34.128435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128345    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d350f3ef-5d12-4021-9fa5-88e397430ecf" containerName="payload-processing"
Oct 30 06:14:34.128435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.128355    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="017b8f8e-f88d-48da-b101-7bd8721647aa" containerName="payload-processing"
Oct 30 06:14:34.133958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.133941    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.151495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.151465    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"]
Oct 30 06:14:34.155315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.155289    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:14:34.155448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.155420    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:14:34.183008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.182969    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfpwv\" (UniqueName: \"kubernetes.io/projected/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-kube-api-access-jfpwv\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.183209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.183033    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.183299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.183266    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.284407 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.284373    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.284762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.284506    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpwv\" (UniqueName: \"kubernetes.io/projected/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-kube-api-access-jfpwv\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.284762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.284530    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.285045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.285017    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.287068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.287044    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.306298 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.306276    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpwv\" (UniqueName: \"kubernetes.io/projected/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-kube-api-access-jfpwv\") pod \"maas-api-revoke-sub-6cdd01b1-wpzrx\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.399045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.399008    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:34.450891 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.450858    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:34.766111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.766077    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"]
Oct 30 06:14:34.767014 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:14:34.766985    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544dc38c_dfe2_4ab3_b3d9_04e44d35f05b.slice/crio-a5154100181ef63f201143c83d230be2103d2bab456ba7b18432fe5244882200 WatchSource:0}: Error finding container a5154100181ef63f201143c83d230be2103d2bab456ba7b18432fe5244882200: Status 404 returned error can't find the container with id a5154100181ef63f201143c83d230be2103d2bab456ba7b18432fe5244882200
Oct 30 06:14:34.826742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:34.826710    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx" event={"ID":"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b","Type":"ContainerStarted","Data":"a5154100181ef63f201143c83d230be2103d2bab456ba7b18432fe5244882200"}
Oct 30 06:14:35.836364 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:35.836335    5250 generic.go:356] "Generic (PLEG): container finished" podID="544dc38c-dfe2-4ab3-b3d9-04e44d35f05b" containerID="0ee76eeec75ca38d1e60dcde4c627dd895911be1ad4c9f3ae16247b126471243" exitCode=0
Oct 30 06:14:35.836791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:35.836385    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx" event={"ID":"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b","Type":"ContainerDied","Data":"0ee76eeec75ca38d1e60dcde4c627dd895911be1ad4c9f3ae16247b126471243"}
Oct 30 06:14:36.147485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:36.147450    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:36.293759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:36.293710    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:36.936118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:36.935069    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"]
Oct 30 06:14:37.017083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.016821    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:37.218973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.218941    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:37.239353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.239320    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-openshift-ca\") pod \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") "
Oct 30 06:14:37.239589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.239570    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-kube-api-access-jfpwv\" (UniqueName: \"kubernetes.io/projected/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-kube-api-access-jfpwv\") pod \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") "
Oct 30 06:14:37.239760 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.239733    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-openshift-ca" pod "544dc38c-dfe2-4ab3-b3d9-04e44d35f05b" (UID: "544dc38c-dfe2-4ab3-b3d9-04e44d35f05b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:14:37.239837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.239816    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-xks-ca\") pod \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\" (UID: \"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b\") "
Oct 30 06:14:37.240401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.240382    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:37.242526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.242497    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-kube-api-access-jfpwv" pod "544dc38c-dfe2-4ab3-b3d9-04e44d35f05b" (UID: "544dc38c-dfe2-4ab3-b3d9-04e44d35f05b"). InnerVolumeSpecName "kube-api-access-jfpwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:14:37.242641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.242551    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-xks-ca" pod "544dc38c-dfe2-4ab3-b3d9-04e44d35f05b" (UID: "544dc38c-dfe2-4ab3-b3d9-04e44d35f05b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:14:37.342347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.341860    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfpwv\" (UniqueName: \"kubernetes.io/projected/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-kube-api-access-jfpwv\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:37.342347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.341895    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:37.540254 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:14:37.540148    5250 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544dc38c_dfe2_4ab3_b3d9_04e44d35f05b.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 06:14:37.575440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.572450    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"]
Oct 30 06:14:37.575440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.573672    5250 cpu_manager.go:414] "containerMap: removing container" podUID="544dc38c-dfe2-4ab3-b3d9-04e44d35f05b" containerName="revoke-keys"
Oct 30 06:14:37.575440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.573692    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544dc38c-dfe2-4ab3-b3d9-04e44d35f05b" containerName="revoke-keys"
Oct 30 06:14:37.575440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.573879    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="544dc38c-dfe2-4ab3-b3d9-04e44d35f05b" containerName="revoke-keys"
Oct 30 06:14:37.579584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.579546    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"]
Oct 30 06:14:37.580058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.580030    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.739770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.739730    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvh8\" (UniqueName: \"kubernetes.io/projected/215db383-2283-430f-b982-c0146fdabb88-kube-api-access-5lvh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.739986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.739921    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215db383-2283-430f-b982-c0146fdabb88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.740251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.740220    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215db383-2283-430f-b982-c0146fdabb88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.842967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.842866    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvh8\" (UniqueName: \"kubernetes.io/projected/215db383-2283-430f-b982-c0146fdabb88-kube-api-access-5lvh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.843158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.842997    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215db383-2283-430f-b982-c0146fdabb88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.843158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.843140    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215db383-2283-430f-b982-c0146fdabb88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.843829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.843804    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215db383-2283-430f-b982-c0146fdabb88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.846241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.846169    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215db383-2283-430f-b982-c0146fdabb88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.861254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.861217    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx" event={"ID":"544dc38c-dfe2-4ab3-b3d9-04e44d35f05b","Type":"ContainerDied","Data":"a5154100181ef63f201143c83d230be2103d2bab456ba7b18432fe5244882200"}
Oct 30 06:14:37.861254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.861252    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"
Oct 30 06:14:37.861486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.861284    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5154100181ef63f201143c83d230be2103d2bab456ba7b18432fe5244882200"
Oct 30 06:14:37.865169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.865136    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvh8\" (UniqueName: \"kubernetes.io/projected/215db383-2283-430f-b982-c0146fdabb88-kube-api-access-5lvh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:37.898783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:37.898741    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:38.127338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:38.127219    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:38.215366 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:38.215340    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"]
Oct 30 06:14:38.216581 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:14:38.216556    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215db383_2283_430f_b982_c0146fdabb88.slice/crio-6cfbadaeca0e392c1515874cba5b4947497627d6c9e672ad5cd04bfa88bcb300 WatchSource:0}: Error finding container 6cfbadaeca0e392c1515874cba5b4947497627d6c9e672ad5cd04bfa88bcb300: Status 404 returned error can't find the container with id 6cfbadaeca0e392c1515874cba5b4947497627d6c9e672ad5cd04bfa88bcb300
Oct 30 06:14:38.873450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:38.873343    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv" event={"ID":"215db383-2283-430f-b982-c0146fdabb88","Type":"ContainerStarted","Data":"49c2de2376e2ef78f5cb9cbb0591f128115e3fd92dd3807c608018843f2c5356"}
Oct 30 06:14:38.873450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:38.873415    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv" event={"ID":"215db383-2283-430f-b982-c0146fdabb88","Type":"ContainerStarted","Data":"6cfbadaeca0e392c1515874cba5b4947497627d6c9e672ad5cd04bfa88bcb300"}
Oct 30 06:14:38.924115 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:38.924065    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv" podStartSLOduration=1.924052187 podStartE2EDuration="1.924052187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:38.919701802 +0000 UTC m=+2593.750739019" watchObservedRunningTime="2026-10-30 06:14:38.924052187 +0000 UTC m=+2593.755089443"
Oct 30 06:14:39.690089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:39.690055    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:39.883348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:39.883308    5250 generic.go:356] "Generic (PLEG): container finished" podID="215db383-2283-430f-b982-c0146fdabb88" containerID="49c2de2376e2ef78f5cb9cbb0591f128115e3fd92dd3807c608018843f2c5356" exitCode=0
Oct 30 06:14:39.883556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:39.883390    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv" event={"ID":"215db383-2283-430f-b982-c0146fdabb88","Type":"ContainerDied","Data":"49c2de2376e2ef78f5cb9cbb0591f128115e3fd92dd3807c608018843f2c5356"}
Oct 30 06:14:41.060330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.060305    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:41.198814 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.198735    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/215db383-2283-430f-b982-c0146fdabb88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215db383-2283-430f-b982-c0146fdabb88-maas-api-openshift-ca\") pod \"215db383-2283-430f-b982-c0146fdabb88\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") "
Oct 30 06:14:41.198963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.198891    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/215db383-2283-430f-b982-c0146fdabb88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215db383-2283-430f-b982-c0146fdabb88-maas-api-xks-ca\") pod \"215db383-2283-430f-b982-c0146fdabb88\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") "
Oct 30 06:14:41.199008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.198984    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215db383-2283-430f-b982-c0146fdabb88-kube-api-access-5lvh8\" (UniqueName: \"kubernetes.io/projected/215db383-2283-430f-b982-c0146fdabb88-kube-api-access-5lvh8\") pod \"215db383-2283-430f-b982-c0146fdabb88\" (UID: \"215db383-2283-430f-b982-c0146fdabb88\") "
Oct 30 06:14:41.199220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.199155    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215db383-2283-430f-b982-c0146fdabb88-maas-api-openshift-ca" pod "215db383-2283-430f-b982-c0146fdabb88" (UID: "215db383-2283-430f-b982-c0146fdabb88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:14:41.199585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.199568    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215db383-2283-430f-b982-c0146fdabb88-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:41.201266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.201240    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215db383-2283-430f-b982-c0146fdabb88-maas-api-xks-ca" pod "215db383-2283-430f-b982-c0146fdabb88" (UID: "215db383-2283-430f-b982-c0146fdabb88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:14:41.201354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.201242    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215db383-2283-430f-b982-c0146fdabb88-kube-api-access-5lvh8" pod "215db383-2283-430f-b982-c0146fdabb88" (UID: "215db383-2283-430f-b982-c0146fdabb88"). InnerVolumeSpecName "kube-api-access-5lvh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:14:41.301023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.300994    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215db383-2283-430f-b982-c0146fdabb88-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:41.301023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.301023    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvh8\" (UniqueName: \"kubernetes.io/projected/215db383-2283-430f-b982-c0146fdabb88-kube-api-access-5lvh8\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:41.507863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.507771    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:41.906856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.906826    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"
Oct 30 06:14:41.907016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.906863    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv" event={"ID":"215db383-2283-430f-b982-c0146fdabb88","Type":"ContainerDied","Data":"6cfbadaeca0e392c1515874cba5b4947497627d6c9e672ad5cd04bfa88bcb300"}
Oct 30 06:14:41.907016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:41.906914    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cfbadaeca0e392c1515874cba5b4947497627d6c9e672ad5cd04bfa88bcb300"
Oct 30 06:14:42.697234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:42.697171    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:42.847206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:42.845707    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:43.774000 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:43.773965    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:44.418254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.418220    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"]
Oct 30 06:14:44.418937 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.418923    5250 cpu_manager.go:414] "containerMap: removing container" podUID="215db383-2283-430f-b982-c0146fdabb88" containerName="revoke-keys"
Oct 30 06:14:44.418977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.418939    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215db383-2283-430f-b982-c0146fdabb88" containerName="revoke-keys"
Oct 30 06:14:44.419064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.419055    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="215db383-2283-430f-b982-c0146fdabb88" containerName="revoke-keys"
Oct 30 06:14:44.422864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.422848    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.435939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.435916    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:14:44.446342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.446314    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:14:44.453608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.453541    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"]
Oct 30 06:14:44.470842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.470815    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.471139 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.471111    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.471279 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.471208    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5r9h\" (UniqueName: \"kubernetes.io/projected/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-kube-api-access-h5r9h\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.572658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.572622    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.572822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.572672    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5r9h\" (UniqueName: \"kubernetes.io/projected/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-kube-api-access-h5r9h\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.572870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.572820    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.573273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.573248    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.575358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.575339    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.594585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.594557    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5r9h\" (UniqueName: \"kubernetes.io/projected/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-kube-api-access-h5r9h\") pod \"maas-api-revoke-sub-388194bc-txnpz\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:44.737981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:44.737902    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:45.047793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:45.047757    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"]
Oct 30 06:14:45.048662 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:14:45.048636    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cb89ee_32a9_4a1e_a8c9_df1c7e9000af.slice/crio-83ecb70e7c9d467269ae280406b375624dba18cbe77b02e8b7d081adbce8f5b5 WatchSource:0}: Error finding container 83ecb70e7c9d467269ae280406b375624dba18cbe77b02e8b7d081adbce8f5b5: Status 404 returned error can't find the container with id 83ecb70e7c9d467269ae280406b375624dba18cbe77b02e8b7d081adbce8f5b5
Oct 30 06:14:45.307824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:45.307725    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:45.944491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:45.944453    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz" event={"ID":"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af","Type":"ContainerStarted","Data":"3ed950be43e5538e064d7683ce0af00163d7a12e26c6c45be862cd3d418c4523"}
Oct 30 06:14:45.944709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:45.944520    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz" event={"ID":"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af","Type":"ContainerStarted","Data":"83ecb70e7c9d467269ae280406b375624dba18cbe77b02e8b7d081adbce8f5b5"}
Oct 30 06:14:45.987887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:45.986973    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz" podStartSLOduration=1.9869543809999999 podStartE2EDuration="1.986954381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:45.980673247 +0000 UTC m=+2600.811710465" watchObservedRunningTime="2026-10-30 06:14:45.986954381 +0000 UTC m=+2600.817991637"
Oct 30 06:14:46.954783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:46.954747    5250 generic.go:356] "Generic (PLEG): container finished" podID="89cb89ee-32a9-4a1e-a8c9-df1c7e9000af" containerID="3ed950be43e5538e064d7683ce0af00163d7a12e26c6c45be862cd3d418c4523" exitCode=0
Oct 30 06:14:46.955240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:46.954823    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz" event={"ID":"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af","Type":"ContainerDied","Data":"3ed950be43e5538e064d7683ce0af00163d7a12e26c6c45be862cd3d418c4523"}
Oct 30 06:14:47.091197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:47.090444    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:47.918002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:47.917966    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:48.069653 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.069606    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:48.169731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.169662    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:48.230935 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.230900    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-xks-ca\") pod \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") "
Oct 30 06:14:48.231106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.231034    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-openshift-ca\") pod \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") "
Oct 30 06:14:48.231339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.231318    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-kube-api-access-h5r9h\" (UniqueName: \"kubernetes.io/projected/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-kube-api-access-h5r9h\") pod \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\" (UID: \"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af\") "
Oct 30 06:14:48.231461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.231438    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-openshift-ca" pod "89cb89ee-32a9-4a1e-a8c9-df1c7e9000af" (UID: "89cb89ee-32a9-4a1e-a8c9-df1c7e9000af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:14:48.232012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.231991    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:48.233396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.233369    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-xks-ca" pod "89cb89ee-32a9-4a1e-a8c9-df1c7e9000af" (UID: "89cb89ee-32a9-4a1e-a8c9-df1c7e9000af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:14:48.233523 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.233507    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-kube-api-access-h5r9h" pod "89cb89ee-32a9-4a1e-a8c9-df1c7e9000af" (UID: "89cb89ee-32a9-4a1e-a8c9-df1c7e9000af"). InnerVolumeSpecName "kube-api-access-h5r9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:14:48.358205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.334679    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5r9h\" (UniqueName: \"kubernetes.io/projected/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-kube-api-access-h5r9h\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:48.358205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.334721    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:48.620054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.619961    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:48.975482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.975447    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz" event={"ID":"89cb89ee-32a9-4a1e-a8c9-df1c7e9000af","Type":"ContainerDied","Data":"83ecb70e7c9d467269ae280406b375624dba18cbe77b02e8b7d081adbce8f5b5"}
Oct 30 06:14:48.975653 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.975506    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83ecb70e7c9d467269ae280406b375624dba18cbe77b02e8b7d081adbce8f5b5"
Oct 30 06:14:48.975653 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.975514    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"
Oct 30 06:14:48.983420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:48.983394    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:49.042686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.042645    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"]
Oct 30 06:14:49.043532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.043511    5250 cpu_manager.go:414] "containerMap: removing container" podUID="89cb89ee-32a9-4a1e-a8c9-df1c7e9000af" containerName="revoke-keys"
Oct 30 06:14:49.043658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.043535    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cb89ee-32a9-4a1e-a8c9-df1c7e9000af" containerName="revoke-keys"
Oct 30 06:14:49.043721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.043690    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cb89ee-32a9-4a1e-a8c9-df1c7e9000af" containerName="revoke-keys"
Oct 30 06:14:49.048156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.048130    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.093024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.092997    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5a591d-openshift-default-dockercfg-w52xb\""
Oct 30 06:14:49.104651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.104624    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"]
Oct 30 06:14:49.150105 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.150061    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158297    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-envoy\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158449    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-data\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158519    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-token\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158576    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-kube-api-access-2lq5x\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158650    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-credential-socket\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158698    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-certs\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158799    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723e1a96-a612-4c85-9dbe-1d71d08938ba-istiod-ca-cert\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158858    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-socket\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.160065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.158886    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-podinfo\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.268111    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-envoy\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.268249    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-data\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.268730    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-token\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.268795    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-kube-api-access-2lq5x\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.268869    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-credential-socket\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.268912    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-certs\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.269008    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723e1a96-a612-4c85-9dbe-1d71d08938ba-istiod-ca-cert\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.269070    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-socket\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.269098    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-podinfo\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.270804    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723e1a96-a612-4c85-9dbe-1d71d08938ba-istiod-ca-cert\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.271093    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-certs\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.271663    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-data\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.272248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.271932    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-socket\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.318231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.315762    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-credential-socket\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.365782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.349738    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:49.365782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.363471    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-podinfo\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.365782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.363826    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-envoy\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.365782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.365720    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-token\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.372200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.370844    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-kube-api-access-2lq5x\") pod \"e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") " pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.372200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.371134    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:49.759287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.759260    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"]
Oct 30 06:14:49.759893 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:14:49.759869    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723e1a96_a612_4c85_9dbe_1d71d08938ba.slice/crio-10d766e3fb3696a065eea6ecda9198916c702025007e5d296460f495d12c9db2 WatchSource:0}: Error finding container 10d766e3fb3696a065eea6ecda9198916c702025007e5d296460f495d12c9db2: Status 404 returned error can't find the container with id 10d766e3fb3696a065eea6ecda9198916c702025007e5d296460f495d12c9db2
Oct 30 06:14:49.762937 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.762903    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:14:49.763055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.762970    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:14:49.763055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.763001    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:14:49.861432 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.861407    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"]
Oct 30 06:14:49.866240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.866212    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.888293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.888255    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"]
Oct 30 06:14:49.987130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987085    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.987324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987205    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62d5387-cd7e-4673-8cd9-e05738171903-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.987324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987264    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.987467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987327    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.987467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987390    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.987562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987485    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.987619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987602    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.987682 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987668    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62d5387-cd7e-4673-8cd9-e05738171903-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:49.987740 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:49.987710    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gpfq\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-kube-api-access-6gpfq\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.000373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.000338    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" event={"ID":"723e1a96-a612-4c85-9dbe-1d71d08938ba","Type":"ContainerStarted","Data":"8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b"}
Oct 30 06:14:50.000537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.000415    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" event={"ID":"723e1a96-a612-4c85-9dbe-1d71d08938ba","Type":"ContainerStarted","Data":"10d766e3fb3696a065eea6ecda9198916c702025007e5d296460f495d12c9db2"}
Oct 30 06:14:50.044043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.043935    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podStartSLOduration=1.043916778 podStartE2EDuration="1.043916778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:50.041534384 +0000 UTC m=+2604.872571638" watchObservedRunningTime="2026-10-30 06:14:50.043916778 +0000 UTC m=+2604.874953997"
Oct 30 06:14:50.088952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.088918    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089019    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089066    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62d5387-cd7e-4673-8cd9-e05738171903-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089099    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gpfq\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-kube-api-access-6gpfq\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089168    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089261    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62d5387-cd7e-4673-8cd9-e05738171903-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089318    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089407    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089474    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089598    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089801    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.089922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089853    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.090019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.089997    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62d5387-cd7e-4673-8cd9-e05738171903-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.090347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.090323    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.092339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.092316    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.092719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.092692    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62d5387-cd7e-4673-8cd9-e05738171903-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.113150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.113116    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gpfq\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-kube-api-access-6gpfq\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.113838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.113815    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.183771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.181617    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:50.379248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.373509    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:50.392086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.392026    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:50.475664 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.475624    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:50.562206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.561498    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"]
Oct 30 06:14:50.566393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.565058    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:14:50.566393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.565141    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:14:50.566393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:50.565220    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:14:51.029476 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.028835    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" event={"ID":"e62d5387-cd7e-4673-8cd9-e05738171903","Type":"ContainerStarted","Data":"9dc038e17ec366f7867b7486d98781ddf58731eb4714706c10aca05328163b53"}
Oct 30 06:14:51.029476 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.028899    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:51.029476 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.028928    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" event={"ID":"e62d5387-cd7e-4673-8cd9-e05738171903","Type":"ContainerStarted","Data":"c8b11ab5f052e6add459c682ec4e993d44dc08d66226b9292f1eb6404cbb48e4"}
Oct 30 06:14:51.033799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.033484    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:14:51.071691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.071565    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podStartSLOduration=2.071547183 podStartE2EDuration="2.071547183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:51.069485021 +0000 UTC m=+2605.900522238" watchObservedRunningTime="2026-10-30 06:14:51.071547183 +0000 UTC m=+2605.902584402"
Oct 30 06:14:51.125205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.124739    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:51.183203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.182571    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:51.183947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.183920    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 06:14:51.184096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.183966    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 06:14:51.267585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.267541    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:51.362631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.362542    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"]
Oct 30 06:14:51.391242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.386582    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-0b481b-mpnvv"]
Oct 30 06:14:51.414086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.414049    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:51.755860 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.755828    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:51.986925 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:51.986876    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p" podUID="082b94fb-4e5e-4739-8616-85ad30f74065" containerName="main" containerID="cri-o://2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91" gracePeriod=45
Oct 30 06:14:52.186578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.185972    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 06:14:52.186578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.186052    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 06:14:52.331797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.331773    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:14:52.486354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486311    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-dshm\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-dshm\") pod \"082b94fb-4e5e-4739-8616-85ad30f74065\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") "
Oct 30 06:14:52.486354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486350    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-model-cache\") pod \"082b94fb-4e5e-4739-8616-85ad30f74065\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") "
Oct 30 06:14:52.486617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486432    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/082b94fb-4e5e-4739-8616-85ad30f74065-tls-certs\" (UniqueName: \"kubernetes.io/secret/082b94fb-4e5e-4739-8616-85ad30f74065-tls-certs\") pod \"082b94fb-4e5e-4739-8616-85ad30f74065\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") "
Oct 30 06:14:52.486617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486467    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-home\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-home\") pod \"082b94fb-4e5e-4739-8616-85ad30f74065\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") "
Oct 30 06:14:52.486617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486525    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-tmp-dir\") pod \"082b94fb-4e5e-4739-8616-85ad30f74065\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") "
Oct 30 06:14:52.486776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486696    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-model-cache" pod "082b94fb-4e5e-4739-8616-85ad30f74065" (UID: "082b94fb-4e5e-4739-8616-85ad30f74065"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:14:52.486776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486706    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082b94fb-4e5e-4739-8616-85ad30f74065-kube-api-access-jf4k2\" (UniqueName: \"kubernetes.io/projected/082b94fb-4e5e-4739-8616-85ad30f74065-kube-api-access-jf4k2\") pod \"082b94fb-4e5e-4739-8616-85ad30f74065\" (UID: \"082b94fb-4e5e-4739-8616-85ad30f74065\") "
Oct 30 06:14:52.486776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486730    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-home" pod "082b94fb-4e5e-4739-8616-85ad30f74065" (UID: "082b94fb-4e5e-4739-8616-85ad30f74065"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:14:52.486943 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.486863    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-tmp-dir" pod "082b94fb-4e5e-4739-8616-85ad30f74065" (UID: "082b94fb-4e5e-4739-8616-85ad30f74065"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:14:52.487733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.487694    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:52.487733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.487724    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:52.487733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.487738    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:52.491223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.489315    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082b94fb-4e5e-4739-8616-85ad30f74065-kube-api-access-jf4k2" pod "082b94fb-4e5e-4739-8616-85ad30f74065" (UID: "082b94fb-4e5e-4739-8616-85ad30f74065"). InnerVolumeSpecName "kube-api-access-jf4k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:14:52.491223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.489418    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-dshm" pod "082b94fb-4e5e-4739-8616-85ad30f74065" (UID: "082b94fb-4e5e-4739-8616-85ad30f74065"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:14:52.491223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.490783    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082b94fb-4e5e-4739-8616-85ad30f74065-tls-certs" pod "082b94fb-4e5e-4739-8616-85ad30f74065" (UID: "082b94fb-4e5e-4739-8616-85ad30f74065"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:14:52.589815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.589213    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4k2\" (UniqueName: \"kubernetes.io/projected/082b94fb-4e5e-4739-8616-85ad30f74065-kube-api-access-jf4k2\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:52.589815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.589250    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082b94fb-4e5e-4739-8616-85ad30f74065-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:52.589815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.589265    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082b94fb-4e5e-4739-8616-85ad30f74065-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:14:52.679204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.677373    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:52.831071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:52.831024    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:53.047786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.047752    5250 generic.go:356] "Generic (PLEG): container finished" podID="082b94fb-4e5e-4739-8616-85ad30f74065" containerID="2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91" exitCode=0
Oct 30 06:14:53.047955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.047802    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"
Oct 30 06:14:53.047955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.047840    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p" event={"ID":"082b94fb-4e5e-4739-8616-85ad30f74065","Type":"ContainerDied","Data":"2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91"}
Oct 30 06:14:53.047955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.047892    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p" event={"ID":"082b94fb-4e5e-4739-8616-85ad30f74065","Type":"ContainerDied","Data":"c493cc8d71ac4fce84b1145355707d2fe0e50fe6d3680a503854b6054c3ca5e9"}
Oct 30 06:14:53.047955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.047915    5250 scope.go:122] "RemoveContainer" containerID="2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91"
Oct 30 06:14:53.085910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.085882    5250 scope.go:122] "RemoveContainer" containerID="2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91"
Oct 30 06:14:53.086303 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:14:53.086281    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91\": container with ID starting with 2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91 not found: ID does not exist" containerID="2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91"
Oct 30 06:14:53.086387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.086313    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91"} err="failed to get container status \"2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91\": rpc error: code = NotFound desc = could not find container \"2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91\": container with ID starting with 2bb776763ade89a41649b6d5c1c3ba8e547ec9fa1211934ed7812ab8ad397c91 not found: ID does not exist"
Oct 30 06:14:53.120900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.120871    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"]
Oct 30 06:14:53.129804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.129779    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b4ca14/auth-test-model-b4ca14-kserve-76d944d8f-tm84p"]
Oct 30 06:14:53.182816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.182778    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 06:14:53.182989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.182831    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 06:14:53.300828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.300429    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:53.315461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.315422    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082b94fb-4e5e-4739-8616-85ad30f74065" path="/var/lib/kubelet/pods/082b94fb-4e5e-4739-8616-85ad30f74065/volumes"
Oct 30 06:14:53.316977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:53.316953    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215db383-2283-430f-b982-c0146fdabb88" path="/var/lib/kubelet/pods/215db383-2283-430f-b982-c0146fdabb88/volumes"
Oct 30 06:14:54.184204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.183476    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 06:14:54.184204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.183540    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 06:14:54.231743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.231710    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:54.309543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.309510    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"]
Oct 30 06:14:54.310271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.310247    5250 cpu_manager.go:414] "containerMap: removing container" podUID="082b94fb-4e5e-4739-8616-85ad30f74065" containerName="main"
Oct 30 06:14:54.310271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.310270    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082b94fb-4e5e-4739-8616-85ad30f74065" containerName="main"
Oct 30 06:14:54.310439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.310428    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="082b94fb-4e5e-4739-8616-85ad30f74065" containerName="main"
Oct 30 06:14:54.315382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.315363    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:54.335404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.335378    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhlt\" (UniqueName: \"kubernetes.io/projected/e9b88e9a-ee31-4222-aff6-5be1b9d89223-kube-api-access-fbhlt\") pod \"payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7\" (UID: \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:54.335654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.335635    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b88e9a-ee31-4222-aff6-5be1b9d89223-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7\" (UID: \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:54.385204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.377533    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5a591d-dockercfg-kspqd\""
Oct 30 06:14:54.388101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.388069    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"]
Oct 30 06:14:54.388465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.388446    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5a591d\""
Oct 30 06:14:54.462259 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.437282    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhlt\" (UniqueName: \"kubernetes.io/projected/e9b88e9a-ee31-4222-aff6-5be1b9d89223-kube-api-access-fbhlt\") pod \"payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7\" (UID: \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:54.462259 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.439262    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b88e9a-ee31-4222-aff6-5be1b9d89223-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7\" (UID: \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:54.462259 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.438023    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b88e9a-ee31-4222-aff6-5be1b9d89223-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7\" (UID: \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:54.547789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.544795    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:54.548791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.548730    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhlt\" (UniqueName: \"kubernetes.io/projected/e9b88e9a-ee31-4222-aff6-5be1b9d89223-kube-api-access-fbhlt\") pod \"payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7\" (UID: \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:54.658756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.644642    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:54.714375 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.714345    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"]
Oct 30 06:14:54.720408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.720383    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"]
Oct 30 06:14:54.720812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.720795    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:54.779204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.770817    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfjk5\" (UniqueName: \"kubernetes.io/projected/710b5db5-0169-4211-bc62-b9beb570471d-kube-api-access-bfjk5\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2\" (UID: \"710b5db5-0169-4211-bc62-b9beb570471d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:54.779204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.770939    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710b5db5-0169-4211-bc62-b9beb570471d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2\" (UID: \"710b5db5-0169-4211-bc62-b9beb570471d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:54.878455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.878410    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfjk5\" (UniqueName: \"kubernetes.io/projected/710b5db5-0169-4211-bc62-b9beb570471d-kube-api-access-bfjk5\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2\" (UID: \"710b5db5-0169-4211-bc62-b9beb570471d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:54.878667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.878530    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710b5db5-0169-4211-bc62-b9beb570471d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2\" (UID: \"710b5db5-0169-4211-bc62-b9beb570471d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:54.883150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.879510    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710b5db5-0169-4211-bc62-b9beb570471d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2\" (UID: \"710b5db5-0169-4211-bc62-b9beb570471d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:54.902273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:54.902251    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfjk5\" (UniqueName: \"kubernetes.io/projected/710b5db5-0169-4211-bc62-b9beb570471d-kube-api-access-bfjk5\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2\" (UID: \"710b5db5-0169-4211-bc62-b9beb570471d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:55.050230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:55.050197    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"]
Oct 30 06:14:55.051684 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:14:55.051657    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b88e9a_ee31_4222_aff6_5be1b9d89223.slice/crio-bc43093e1f16016470c3b187f43c8b57b199d884ad2d11a8bedc9b23bac6315b WatchSource:0}: Error finding container bc43093e1f16016470c3b187f43c8b57b199d884ad2d11a8bedc9b23bac6315b: Status 404 returned error can't find the container with id bc43093e1f16016470c3b187f43c8b57b199d884ad2d11a8bedc9b23bac6315b
Oct 30 06:14:55.088458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:55.088431    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:55.095827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:55.095794    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7" event={"ID":"e9b88e9a-ee31-4222-aff6-5be1b9d89223","Type":"ContainerStarted","Data":"bc43093e1f16016470c3b187f43c8b57b199d884ad2d11a8bedc9b23bac6315b"}
Oct 30 06:14:55.185451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:55.183533    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 06:14:55.185451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:55.183578    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 06:14:55.312965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:55.310474    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:55.442914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:55.442885    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"]
Oct 30 06:14:55.443192 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:14:55.443154    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710b5db5_0169_4211_bc62_b9beb570471d.slice/crio-a732bb57f0e105d9086f23b9b0607ad032272013102f2f2428c2ea803173c860 WatchSource:0}: Error finding container a732bb57f0e105d9086f23b9b0607ad032272013102f2f2428c2ea803173c860: Status 404 returned error can't find the container with id a732bb57f0e105d9086f23b9b0607ad032272013102f2f2428c2ea803173c860
Oct 30 06:14:56.105773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.105736    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7" event={"ID":"e9b88e9a-ee31-4222-aff6-5be1b9d89223","Type":"ContainerStarted","Data":"0fd89ee66aa98c3e8411b8d9f33f0586ab035f678a75bc1b825b87d7bf6709aa"}
Oct 30 06:14:56.106001 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.105949    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:14:56.107751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.107721    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2" event={"ID":"710b5db5-0169-4211-bc62-b9beb570471d","Type":"ContainerStarted","Data":"19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110"}
Oct 30 06:14:56.107894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.107765    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2" event={"ID":"710b5db5-0169-4211-bc62-b9beb570471d","Type":"ContainerStarted","Data":"a732bb57f0e105d9086f23b9b0607ad032272013102f2f2428c2ea803173c860"}
Oct 30 06:14:56.107894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.107785    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:14:56.144071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.144011    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7" podStartSLOduration=2.143998789 podStartE2EDuration="2.143998789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:56.142583413 +0000 UTC m=+2610.973620641" watchObservedRunningTime="2026-10-30 06:14:56.143998789 +0000 UTC m=+2610.975036005"
Oct 30 06:14:56.182899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.182861    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 06:14:56.183089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.182932    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 06:14:56.200358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.200308    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2" podStartSLOduration=2.200294932 podStartE2EDuration="2.200294932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:56.19686078 +0000 UTC m=+2611.027897997" watchObservedRunningTime="2026-10-30 06:14:56.200294932 +0000 UTC m=+2611.031332145"
Oct 30 06:14:56.996619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:56.996578    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:57.183211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:57.183160    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 06:14:57.183400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:57.183239    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 06:14:58.079385 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:58.079346    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:58.187893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:58.187866    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:58.188524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:58.188488    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:58.189294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:58.189278    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:14:58.229910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:58.229873    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:58.375785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:58.375696    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"]
Oct 30 06:14:58.375946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:58.375904    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy" containerID="cri-o://ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501" gracePeriod=30
Oct 30 06:14:58.429379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:58.429345    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:59.001858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.001824    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"]
Oct 30 06:14:59.006235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.006213    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.025661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.024287    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"]
Oct 30 06:14:59.028988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.028956    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:14:59.029106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.029020    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:14:59.078297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.078261    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.078577 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.078554    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.078739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.078713    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddxr\" (UniqueName: \"kubernetes.io/projected/4a0f257f-bf55-40dc-81b1-af99c919362a-kube-api-access-zddxr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.180059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.180026    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.180692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.180136    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddxr\" (UniqueName: \"kubernetes.io/projected/4a0f257f-bf55-40dc-81b1-af99c919362a-kube-api-access-zddxr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.180692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.180251    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.180901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.180877    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.182979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.182957    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.201959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.201932    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddxr\" (UniqueName: \"kubernetes.io/projected/4a0f257f-bf55-40dc-81b1-af99c919362a-kube-api-access-zddxr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.322317 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.322241    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:14:59.341086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.341056    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:14:59.751066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.751036    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"]
Oct 30 06:14:59.752195 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:14:59.752143    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0f257f_bf55_40dc_81b1_af99c919362a.slice/crio-0548e7ed4351df0e8944435355fa113f8ca244385aa4447b8532c152b073081e WatchSource:0}: Error finding container 0548e7ed4351df0e8944435355fa113f8ca244385aa4447b8532c152b073081e: Status 404 returned error can't find the container with id 0548e7ed4351df0e8944435355fa113f8ca244385aa4447b8532c152b073081e
Oct 30 06:14:59.933928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:14:59.933895    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:00.145547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.145467    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"]
Oct 30 06:15:00.153423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.153386    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2" event={"ID":"4a0f257f-bf55-40dc-81b1-af99c919362a","Type":"ContainerStarted","Data":"0548e7ed4351df0e8944435355fa113f8ca244385aa4447b8532c152b073081e"}
Oct 30 06:15:00.153600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.153502    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:00.154766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.154743    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"]
Oct 30 06:15:00.159671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.159645    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:00.159671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.159665    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"]
Oct 30 06:15:00.164711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.164679    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:00.172870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.172847    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"]
Oct 30 06:15:00.192417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.191927    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"]
Oct 30 06:15:00.200083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.200057    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"]
Oct 30 06:15:00.204776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.204747    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb\" (UID: \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:00.205507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.205473    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/404e0b49-889c-4102-91fd-b989dcd5445c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw\" (UID: \"404e0b49-889c-4102-91fd-b989dcd5445c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:00.205628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.205532    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwv7\" (UniqueName: \"kubernetes.io/projected/404e0b49-889c-4102-91fd-b989dcd5445c-kube-api-access-mxwv7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw\" (UID: \"404e0b49-889c-4102-91fd-b989dcd5445c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:00.205628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.205579    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsb7n\" (UniqueName: \"kubernetes.io/projected/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-kube-api-access-fsb7n\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb\" (UID: \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:00.206124 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.206099    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"]
Oct 30 06:15:00.227380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.224932    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"]
Oct 30 06:15:00.227380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.225431    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:00.238257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.235883    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:00.262200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.245070    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"]
Oct 30 06:15:00.262200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.252648    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"]
Oct 30 06:15:00.262200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.256350    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:00.268423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.267474    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"]
Oct 30 06:15:00.268423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.267515    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"]
Oct 30 06:15:00.268423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.267608    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:00.268655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.268450    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"]
Oct 30 06:15:00.276701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.276682    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"]
Oct 30 06:15:00.294551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.294528    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"]
Oct 30 06:15:00.295011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.294973    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"]
Oct 30 06:15:00.295337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.294919    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:00.305377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.305358    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336470    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/404e0b49-889c-4102-91fd-b989dcd5445c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw\" (UID: \"404e0b49-889c-4102-91fd-b989dcd5445c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336528    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d2145-c914-4668-b07d-c7058b2d3f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h\" (UID: \"8e3d2145-c914-4668-b07d-c7058b2d3f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336572    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwv7\" (UniqueName: \"kubernetes.io/projected/404e0b49-889c-4102-91fd-b989dcd5445c-kube-api-access-mxwv7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw\" (UID: \"404e0b49-889c-4102-91fd-b989dcd5445c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336615    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsb7n\" (UniqueName: \"kubernetes.io/projected/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-kube-api-access-fsb7n\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb\" (UID: \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336675    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/259369fd-d911-4b38-96d2-ee3b43d3c075-kube-api-access-rxxjf\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2\" (UID: \"259369fd-d911-4b38-96d2-ee3b43d3c075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336722    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01694ac-ebec-4c70-a10a-39f4de159d25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9\" (UID: \"d01694ac-ebec-4c70-a10a-39f4de159d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336760    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb\" (UID: \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336791    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2ws\" (UniqueName: \"kubernetes.io/projected/6fd00ce8-6874-4369-9022-1d1001fcbae3-kube-api-access-xd2ws\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b\" (UID: \"6fd00ce8-6874-4369-9022-1d1001fcbae3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336910    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2dnm\" (UniqueName: \"kubernetes.io/projected/d01694ac-ebec-4c70-a10a-39f4de159d25-kube-api-access-s2dnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9\" (UID: \"d01694ac-ebec-4c70-a10a-39f4de159d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.336999    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6gz\" (UniqueName: \"kubernetes.io/projected/8e3d2145-c914-4668-b07d-c7058b2d3f59-kube-api-access-tv6gz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h\" (UID: \"8e3d2145-c914-4668-b07d-c7058b2d3f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.337057    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259369fd-d911-4b38-96d2-ee3b43d3c075-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2\" (UID: \"259369fd-d911-4b38-96d2-ee3b43d3c075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.337218    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd00ce8-6874-4369-9022-1d1001fcbae3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b\" (UID: \"6fd00ce8-6874-4369-9022-1d1001fcbae3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:00.368068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.338067    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/404e0b49-889c-4102-91fd-b989dcd5445c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw\" (UID: \"404e0b49-889c-4102-91fd-b989dcd5445c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:00.374593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.374564    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb\" (UID: \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:00.374942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.374918    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsb7n\" (UniqueName: \"kubernetes.io/projected/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-kube-api-access-fsb7n\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb\" (UID: \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:00.384359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.380412    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwv7\" (UniqueName: \"kubernetes.io/projected/404e0b49-889c-4102-91fd-b989dcd5445c-kube-api-access-mxwv7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw\" (UID: \"404e0b49-889c-4102-91fd-b989dcd5445c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:00.384359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.381028    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"]
Oct 30 06:15:00.404548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.404446    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"]
Oct 30 06:15:00.404548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.404500    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"]
Oct 30 06:15:00.405163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.405141    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:00.415748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.415535    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"]
Oct 30 06:15:00.417316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.416514    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:00.430391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.430368    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"]
Oct 30 06:15:00.431380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.431358    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:00.440634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.440604    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75km\" (UniqueName: \"kubernetes.io/projected/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-kube-api-access-s75km\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9\" (UID: \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:00.440728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.440658    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/259369fd-d911-4b38-96d2-ee3b43d3c075-kube-api-access-rxxjf\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2\" (UID: \"259369fd-d911-4b38-96d2-ee3b43d3c075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:00.440728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.440702    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01694ac-ebec-4c70-a10a-39f4de159d25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9\" (UID: \"d01694ac-ebec-4c70-a10a-39f4de159d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:00.440842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.440747    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2ws\" (UniqueName: \"kubernetes.io/projected/6fd00ce8-6874-4369-9022-1d1001fcbae3-kube-api-access-xd2ws\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b\" (UID: \"6fd00ce8-6874-4369-9022-1d1001fcbae3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:00.440900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.440873    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfb55-80c6-4651-b46c-e588b914c30c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px\" (UID: \"27ebfb55-80c6-4651-b46c-e588b914c30c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:00.440956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.440933    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2dnm\" (UniqueName: \"kubernetes.io/projected/d01694ac-ebec-4c70-a10a-39f4de159d25-kube-api-access-s2dnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9\" (UID: \"d01694ac-ebec-4c70-a10a-39f4de159d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:00.441051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.441033    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6gz\" (UniqueName: \"kubernetes.io/projected/8e3d2145-c914-4668-b07d-c7058b2d3f59-kube-api-access-tv6gz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h\" (UID: \"8e3d2145-c914-4668-b07d-c7058b2d3f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:00.441116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.441102    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259369fd-d911-4b38-96d2-ee3b43d3c075-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2\" (UID: \"259369fd-d911-4b38-96d2-ee3b43d3c075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:00.441200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.441132    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62622d-84ad-4903-9be4-3b6ec51291f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-rrn6s\" (UID: \"ab62622d-84ad-4903-9be4-3b6ec51291f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:00.441349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.441313    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9\" (UID: \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:00.441424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.441348    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfph\" (UniqueName: \"kubernetes.io/projected/27ebfb55-80c6-4651-b46c-e588b914c30c-kube-api-access-jwfph\") pod \"maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px\" (UID: \"27ebfb55-80c6-4651-b46c-e588b914c30c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:00.441424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.441375    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd00ce8-6874-4369-9022-1d1001fcbae3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b\" (UID: \"6fd00ce8-6874-4369-9022-1d1001fcbae3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:00.443096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.443072    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01694ac-ebec-4c70-a10a-39f4de159d25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9\" (UID: \"d01694ac-ebec-4c70-a10a-39f4de159d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:00.444630 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.444597    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259369fd-d911-4b38-96d2-ee3b43d3c075-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2\" (UID: \"259369fd-d911-4b38-96d2-ee3b43d3c075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:00.446211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.446171    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd00ce8-6874-4369-9022-1d1001fcbae3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b\" (UID: \"6fd00ce8-6874-4369-9022-1d1001fcbae3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:00.446333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.441405    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbv8l\" (UniqueName: \"kubernetes.io/projected/ab62622d-84ad-4903-9be4-3b6ec51291f3-kube-api-access-fbv8l\") pod \"maas-api-key-cleanup-29889015-rrn6s\" (UID: \"ab62622d-84ad-4903-9be4-3b6ec51291f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:00.446442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.446422    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d2145-c914-4668-b07d-c7058b2d3f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h\" (UID: \"8e3d2145-c914-4668-b07d-c7058b2d3f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:00.447143 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.447121    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"]
Oct 30 06:15:00.447271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.447255    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:00.447349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.447323    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d2145-c914-4668-b07d-c7058b2d3f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h\" (UID: \"8e3d2145-c914-4668-b07d-c7058b2d3f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:00.457407 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.457382    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"]
Oct 30 06:15:00.464699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.464674    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/259369fd-d911-4b38-96d2-ee3b43d3c075-kube-api-access-rxxjf\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2\" (UID: \"259369fd-d911-4b38-96d2-ee3b43d3c075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:00.472023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.471834    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"]
Oct 30 06:15:00.479137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.479113    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2ws\" (UniqueName: \"kubernetes.io/projected/6fd00ce8-6874-4369-9022-1d1001fcbae3-kube-api-access-xd2ws\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b\" (UID: \"6fd00ce8-6874-4369-9022-1d1001fcbae3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:00.481342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.481293    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"]
Oct 30 06:15:00.487838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.487817    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:00.496236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.494933    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"]
Oct 30 06:15:00.506747 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.506717    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:00.508314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.508288    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6gz\" (UniqueName: \"kubernetes.io/projected/8e3d2145-c914-4668-b07d-c7058b2d3f59-kube-api-access-tv6gz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h\" (UID: \"8e3d2145-c914-4668-b07d-c7058b2d3f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:00.516526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.516277    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2dnm\" (UniqueName: \"kubernetes.io/projected/d01694ac-ebec-4c70-a10a-39f4de159d25-kube-api-access-s2dnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9\" (UID: \"d01694ac-ebec-4c70-a10a-39f4de159d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:00.520874 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.520851    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:00.541345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.541317    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:00.548439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548411    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlfc\" (UniqueName: \"kubernetes.io/projected/bc73291b-82f1-4261-bc0b-8712d8a49558-kube-api-access-vdlfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p\" (UID: \"bc73291b-82f1-4261-bc0b-8712d8a49558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:00.548560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548468    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75km\" (UniqueName: \"kubernetes.io/projected/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-kube-api-access-s75km\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9\" (UID: \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:00.548560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548489    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88gj\" (UniqueName: \"kubernetes.io/projected/d09445ae-5f09-4c18-a37e-3af42038cd1e-kube-api-access-k88gj\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9\" (UID: \"d09445ae-5f09-4c18-a37e-3af42038cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:00.548560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548507    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5783c3-b770-4ded-a301-7e607e09bdb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq\" (UID: \"4c5783c3-b770-4ded-a301-7e607e09bdb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:00.548560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548542    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9ff\" (UniqueName: \"kubernetes.io/projected/4c5783c3-b770-4ded-a301-7e607e09bdb4-kube-api-access-8p9ff\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq\" (UID: \"4c5783c3-b770-4ded-a301-7e607e09bdb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:00.548812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548601    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfb55-80c6-4651-b46c-e588b914c30c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px\" (UID: \"27ebfb55-80c6-4651-b46c-e588b914c30c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:00.548812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548652    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc73291b-82f1-4261-bc0b-8712d8a49558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p\" (UID: \"bc73291b-82f1-4261-bc0b-8712d8a49558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:00.548812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548719    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62622d-84ad-4903-9be4-3b6ec51291f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-rrn6s\" (UID: \"ab62622d-84ad-4903-9be4-3b6ec51291f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:00.548812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548796    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09445ae-5f09-4c18-a37e-3af42038cd1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9\" (UID: \"d09445ae-5f09-4c18-a37e-3af42038cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:00.549038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548878    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff353edf-ad97-4352-a8b2-470fa965104c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6\" (UID: \"ff353edf-ad97-4352-a8b2-470fa965104c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:00.549038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548895    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9n5\" (UniqueName: \"kubernetes.io/projected/ff353edf-ad97-4352-a8b2-470fa965104c-kube-api-access-6z9n5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6\" (UID: \"ff353edf-ad97-4352-a8b2-470fa965104c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:00.549038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548913    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9\" (UID: \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:00.549038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548931    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfph\" (UniqueName: \"kubernetes.io/projected/27ebfb55-80c6-4651-b46c-e588b914c30c-kube-api-access-jwfph\") pod \"maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px\" (UID: \"27ebfb55-80c6-4651-b46c-e588b914c30c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:00.549038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.548954    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbv8l\" (UniqueName: \"kubernetes.io/projected/ab62622d-84ad-4903-9be4-3b6ec51291f3-kube-api-access-fbv8l\") pod \"maas-api-key-cleanup-29889015-rrn6s\" (UID: \"ab62622d-84ad-4903-9be4-3b6ec51291f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:00.550392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.550039    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9\" (UID: \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:00.550392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.550047    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62622d-84ad-4903-9be4-3b6ec51291f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-rrn6s\" (UID: \"ab62622d-84ad-4903-9be4-3b6ec51291f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:00.550750 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.550713    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfb55-80c6-4651-b46c-e588b914c30c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px\" (UID: \"27ebfb55-80c6-4651-b46c-e588b914c30c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:00.567583 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.564818    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:00.592501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.583629    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75km\" (UniqueName: \"kubernetes.io/projected/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-kube-api-access-s75km\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9\" (UID: \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:00.592501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.583809    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:00.592501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.584067    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfph\" (UniqueName: \"kubernetes.io/projected/27ebfb55-80c6-4651-b46c-e588b914c30c-kube-api-access-jwfph\") pod \"maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px\" (UID: \"27ebfb55-80c6-4651-b46c-e588b914c30c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:00.592501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.588537    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbv8l\" (UniqueName: \"kubernetes.io/projected/ab62622d-84ad-4903-9be4-3b6ec51291f3-kube-api-access-fbv8l\") pod \"maas-api-key-cleanup-29889015-rrn6s\" (UID: \"ab62622d-84ad-4903-9be4-3b6ec51291f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.599842    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.706998    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.712293    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.771534    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.787197    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9ff\" (UniqueName: \"kubernetes.io/projected/4c5783c3-b770-4ded-a301-7e607e09bdb4-kube-api-access-8p9ff\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq\" (UID: \"4c5783c3-b770-4ded-a301-7e607e09bdb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.787380    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc73291b-82f1-4261-bc0b-8712d8a49558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p\" (UID: \"bc73291b-82f1-4261-bc0b-8712d8a49558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.787509    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09445ae-5f09-4c18-a37e-3af42038cd1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9\" (UID: \"d09445ae-5f09-4c18-a37e-3af42038cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.787624    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff353edf-ad97-4352-a8b2-470fa965104c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6\" (UID: \"ff353edf-ad97-4352-a8b2-470fa965104c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.787657    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9n5\" (UniqueName: \"kubernetes.io/projected/ff353edf-ad97-4352-a8b2-470fa965104c-kube-api-access-6z9n5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6\" (UID: \"ff353edf-ad97-4352-a8b2-470fa965104c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.787785    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlfc\" (UniqueName: \"kubernetes.io/projected/bc73291b-82f1-4261-bc0b-8712d8a49558-kube-api-access-vdlfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p\" (UID: \"bc73291b-82f1-4261-bc0b-8712d8a49558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.787843    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88gj\" (UniqueName: \"kubernetes.io/projected/d09445ae-5f09-4c18-a37e-3af42038cd1e-kube-api-access-k88gj\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9\" (UID: \"d09445ae-5f09-4c18-a37e-3af42038cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.787873    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5783c3-b770-4ded-a301-7e607e09bdb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq\" (UID: \"4c5783c3-b770-4ded-a301-7e607e09bdb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.788593    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc73291b-82f1-4261-bc0b-8712d8a49558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p\" (UID: \"bc73291b-82f1-4261-bc0b-8712d8a49558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.790436    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09445ae-5f09-4c18-a37e-3af42038cd1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9\" (UID: \"d09445ae-5f09-4c18-a37e-3af42038cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:00.837772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.790987    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff353edf-ad97-4352-a8b2-470fa965104c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6\" (UID: \"ff353edf-ad97-4352-a8b2-470fa965104c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:00.860101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.838914    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:00.860101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.854515    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5783c3-b770-4ded-a301-7e607e09bdb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq\" (UID: \"4c5783c3-b770-4ded-a301-7e607e09bdb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:00.931899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.910048    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlfc\" (UniqueName: \"kubernetes.io/projected/bc73291b-82f1-4261-bc0b-8712d8a49558-kube-api-access-vdlfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p\" (UID: \"bc73291b-82f1-4261-bc0b-8712d8a49558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:00.931899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.911429    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9n5\" (UniqueName: \"kubernetes.io/projected/ff353edf-ad97-4352-a8b2-470fa965104c-kube-api-access-6z9n5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6\" (UID: \"ff353edf-ad97-4352-a8b2-470fa965104c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:00.933509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.932405    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9ff\" (UniqueName: \"kubernetes.io/projected/4c5783c3-b770-4ded-a301-7e607e09bdb4-kube-api-access-8p9ff\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq\" (UID: \"4c5783c3-b770-4ded-a301-7e607e09bdb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:00.952847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:00.947687    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88gj\" (UniqueName: \"kubernetes.io/projected/d09445ae-5f09-4c18-a37e-3af42038cd1e-kube-api-access-k88gj\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9\" (UID: \"d09445ae-5f09-4c18-a37e-3af42038cd1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:01.078929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.033814    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:01.088424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.079448    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:01.121212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.121146    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:01.167219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.166472    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:01.363656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.363623    5250 generic.go:356] "Generic (PLEG): container finished" podID="4a0f257f-bf55-40dc-81b1-af99c919362a" containerID="759dbcc523bd0c0335b31eb6fa0e74e5a7fb78b357bed857ade29c9a211a766d" exitCode=0
Oct 30 06:15:01.364232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.364204    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2" event={"ID":"4a0f257f-bf55-40dc-81b1-af99c919362a","Type":"ContainerDied","Data":"759dbcc523bd0c0335b31eb6fa0e74e5a7fb78b357bed857ade29c9a211a766d"}
Oct 30 06:15:01.479258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.479205    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"]
Oct 30 06:15:01.645791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.645742    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"]
Oct 30 06:15:01.824017 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.823984    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"]
Oct 30 06:15:01.839416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.839367    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"]
Oct 30 06:15:01.879954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.879927    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"]
Oct 30 06:15:01.978422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:01.977266    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:02.028236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.027489    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"]
Oct 30 06:15:02.028511 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:02.028347    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c63877_7c54_427d_b8bf_c0f1b22dcebb.slice/crio-bf2b5741880a268b03e2eaa74776516faf20fbb2a90ac23e30b198ea23e6acdf WatchSource:0}: Error finding container bf2b5741880a268b03e2eaa74776516faf20fbb2a90ac23e30b198ea23e6acdf: Status 404 returned error can't find the container with id bf2b5741880a268b03e2eaa74776516faf20fbb2a90ac23e30b198ea23e6acdf
Oct 30 06:15:02.048601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.048263    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"]
Oct 30 06:15:02.061055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.061011    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"]
Oct 30 06:15:02.088789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.088757    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"]
Oct 30 06:15:02.127559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.127531    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"]
Oct 30 06:15:02.166847 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:02.166818    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ebfb55_80c6_4651_b46c_e588b914c30c.slice/crio-1d4bc9e451eda2931f26cf46b3641b84776681ec22ab35197ccf2df7157f1ab0 WatchSource:0}: Error finding container 1d4bc9e451eda2931f26cf46b3641b84776681ec22ab35197ccf2df7157f1ab0: Status 404 returned error can't find the container with id 1d4bc9e451eda2931f26cf46b3641b84776681ec22ab35197ccf2df7157f1ab0
Oct 30 06:15:02.229693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.229653    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"]
Oct 30 06:15:02.235119 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.235087    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"]
Oct 30 06:15:02.257413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.257384    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"]
Oct 30 06:15:02.358788 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:02.358753    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc73291b_82f1_4261_bc0b_8712d8a49558.slice/crio-6abac56512a3a36e4bec2c88031d760a1fe22b4ac9dbd54d4d3adf5b55a0a87a WatchSource:0}: Error finding container 6abac56512a3a36e4bec2c88031d760a1fe22b4ac9dbd54d4d3adf5b55a0a87a: Status 404 returned error can't find the container with id 6abac56512a3a36e4bec2c88031d760a1fe22b4ac9dbd54d4d3adf5b55a0a87a
Oct 30 06:15:02.361596 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:02.361570    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5783c3_b770_4ded_a301_7e607e09bdb4.slice/crio-a3986e4db0bfae6031e9405a13362f1f820ba1df01c87144c1e680fda3407105 WatchSource:0}: Error finding container a3986e4db0bfae6031e9405a13362f1f820ba1df01c87144c1e680fda3407105: Status 404 returned error can't find the container with id a3986e4db0bfae6031e9405a13362f1f820ba1df01c87144c1e680fda3407105
Oct 30 06:15:02.376574 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.376492    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" event={"ID":"27ebfb55-80c6-4651-b46c-e588b914c30c","Type":"ContainerStarted","Data":"1d4bc9e451eda2931f26cf46b3641b84776681ec22ab35197ccf2df7157f1ab0"}
Oct 30 06:15:02.378231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.378206    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" event={"ID":"d01694ac-ebec-4c70-a10a-39f4de159d25","Type":"ContainerStarted","Data":"e5a5379f1c80b70bf4999f07e37709fbd8fba6682cf1cf22b25eefe183b9a987"}
Oct 30 06:15:02.379779 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.379758    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" event={"ID":"8e3d2145-c914-4668-b07d-c7058b2d3f59","Type":"ContainerStarted","Data":"db56b0fefe04c9dc4da62da02ce7a91986d17ab338926ab5b2975bda133dfe32"}
Oct 30 06:15:02.381495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.381472    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" event={"ID":"4c5783c3-b770-4ded-a301-7e607e09bdb4","Type":"ContainerStarted","Data":"a3986e4db0bfae6031e9405a13362f1f820ba1df01c87144c1e680fda3407105"}
Oct 30 06:15:02.383043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.383017    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" event={"ID":"6fd00ce8-6874-4369-9022-1d1001fcbae3","Type":"ContainerStarted","Data":"da3e587970c92f79c5f00adadb7907c56c0528bbc9b4a7281d617dcf8a985571"}
Oct 30 06:15:02.385494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.384764    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" event={"ID":"404e0b49-889c-4102-91fd-b989dcd5445c","Type":"ContainerStarted","Data":"b84277bf73d0c709251d57c6a35f17c4ff9ff073ef7874be95c3ae400a70ba18"}
Oct 30 06:15:02.385494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.384806    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" event={"ID":"404e0b49-889c-4102-91fd-b989dcd5445c","Type":"ContainerStarted","Data":"3f9a81516c5fc76bf4c7d49e0b01f97dde26170726149f2141be78a83f0e54bf"}
Oct 30 06:15:02.386991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.386960    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" event={"ID":"ff353edf-ad97-4352-a8b2-470fa965104c","Type":"ContainerStarted","Data":"b5a801db38cc7790649a513234097123e985d92690982a713698372a1fe8d00b"}
Oct 30 06:15:02.388560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.388537    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" event={"ID":"259369fd-d911-4b38-96d2-ee3b43d3c075","Type":"ContainerStarted","Data":"0fe63dec3b003dcc3eb3b64e7518606745ac80bc7b28be70c489d583a1044e1d"}
Oct 30 06:15:02.389845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.389823    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" event={"ID":"c9c63877-7c54-427d-b8bf-c0f1b22dcebb","Type":"ContainerStarted","Data":"bf2b5741880a268b03e2eaa74776516faf20fbb2a90ac23e30b198ea23e6acdf"}
Oct 30 06:15:02.391030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.390998    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" event={"ID":"bc73291b-82f1-4261-bc0b-8712d8a49558","Type":"ContainerStarted","Data":"6abac56512a3a36e4bec2c88031d760a1fe22b4ac9dbd54d4d3adf5b55a0a87a"}
Oct 30 06:15:02.392394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.392373    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" event={"ID":"d09445ae-5f09-4c18-a37e-3af42038cd1e","Type":"ContainerStarted","Data":"634ef52a7cb0792e70522ce848b1ee6dbcc7e02bc0b1fe8273fc73b02c28c1b3"}
Oct 30 06:15:02.393754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.393731    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" event={"ID":"ab62622d-84ad-4903-9be4-3b6ec51291f3","Type":"ContainerStarted","Data":"e5c0165f66200ffed88f262333df5d4b7c5d7e0c128bfe3f1e2e13ec609894a2"}
Oct 30 06:15:02.395384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.395360    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" event={"ID":"f3be9b16-06d1-4ab5-a8e8-1825afb98f39","Type":"ContainerStarted","Data":"38b0fee18e1e90b17f3ec780ec7061e82ba18624d0f693b6679dba5a8ce931e4"}
Oct 30 06:15:02.474844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.474785    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" podStartSLOduration=2.474768252 podStartE2EDuration="2.474768252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.426288217 +0000 UTC m=+2617.257325445" watchObservedRunningTime="2026-10-30 06:15:02.474768252 +0000 UTC m=+2617.305805469"
Oct 30 06:15:02.602869 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:02.602369    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:15:02.936296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.936228    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:15:02.982365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:02.977958    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" podStartSLOduration=2.977938979 podStartE2EDuration="2.977938979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.469477867 +0000 UTC m=+2617.300515086" watchObservedRunningTime="2026-10-30 06:15:02.977938979 +0000 UTC m=+2617.808976197"
Oct 30 06:15:03.004503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.002918    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0f257f-bf55-40dc-81b1-af99c919362a-kube-api-access-zddxr\" (UniqueName: \"kubernetes.io/projected/4a0f257f-bf55-40dc-81b1-af99c919362a-kube-api-access-zddxr\") pod \"4a0f257f-bf55-40dc-81b1-af99c919362a\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") "
Oct 30 06:15:03.004503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.003327    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-openshift-ca\") pod \"4a0f257f-bf55-40dc-81b1-af99c919362a\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") "
Oct 30 06:15:03.004503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.003414    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-xks-ca\") pod \"4a0f257f-bf55-40dc-81b1-af99c919362a\" (UID: \"4a0f257f-bf55-40dc-81b1-af99c919362a\") "
Oct 30 06:15:03.005975 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.005949    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-openshift-ca" pod "4a0f257f-bf55-40dc-81b1-af99c919362a" (UID: "4a0f257f-bf55-40dc-81b1-af99c919362a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:03.007494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.007440    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-xks-ca" pod "4a0f257f-bf55-40dc-81b1-af99c919362a" (UID: "4a0f257f-bf55-40dc-81b1-af99c919362a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:15:03.009808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.009772    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0f257f-bf55-40dc-81b1-af99c919362a-kube-api-access-zddxr" pod "4a0f257f-bf55-40dc-81b1-af99c919362a" (UID: "4a0f257f-bf55-40dc-81b1-af99c919362a"). InnerVolumeSpecName "kube-api-access-zddxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:03.106562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.106521    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:03.106562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.106564    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0f257f-bf55-40dc-81b1-af99c919362a-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:03.106820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.106580    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddxr\" (UniqueName: \"kubernetes.io/projected/4a0f257f-bf55-40dc-81b1-af99c919362a-kube-api-access-zddxr\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:03.429347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.429307    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" event={"ID":"27ebfb55-80c6-4651-b46c-e588b914c30c","Type":"ContainerStarted","Data":"40b4d006c0c48e86ba8d1508df9b6652e9346b161878c8e0e813f83520f719bf"}
Oct 30 06:15:03.431413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.431383    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" event={"ID":"d01694ac-ebec-4c70-a10a-39f4de159d25","Type":"ContainerStarted","Data":"7dee05fb3f20f56a8d8e933d7f2c24e6c4231cc14e2070726abdb059fae74ac1"}
Oct 30 06:15:03.433540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.433513    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" event={"ID":"8e3d2145-c914-4668-b07d-c7058b2d3f59","Type":"ContainerStarted","Data":"4574b1bea32b038e506c864b0bedefd10f272b91d1d1751f9af45c43f8b9ed05"}
Oct 30 06:15:03.437568 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.437532    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" event={"ID":"4c5783c3-b770-4ded-a301-7e607e09bdb4","Type":"ContainerStarted","Data":"b20934a2c31aa2c8c4399f0a4af8e58d3fe61ec32852b5a785481642416059a8"}
Oct 30 06:15:03.443448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.443425    5250 generic.go:356] "Generic (PLEG): container finished" podID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerID="e7217eee0e87cb5fd198edaa930dd800edd95e86a62a3e7ebd84a7bc334e79cb" exitCode=60
Oct 30 06:15:03.443663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.443638    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" event={"ID":"6fd00ce8-6874-4369-9022-1d1001fcbae3","Type":"ContainerDied","Data":"e7217eee0e87cb5fd198edaa930dd800edd95e86a62a3e7ebd84a7bc334e79cb"}
Oct 30 06:15:03.443964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.443945    5250 scope.go:122] "RemoveContainer" containerID="e7217eee0e87cb5fd198edaa930dd800edd95e86a62a3e7ebd84a7bc334e79cb"
Oct 30 06:15:03.454051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.454009    5250 generic.go:356] "Generic (PLEG): container finished" podID="404e0b49-889c-4102-91fd-b989dcd5445c" containerID="b84277bf73d0c709251d57c6a35f17c4ff9ff073ef7874be95c3ae400a70ba18" exitCode=60
Oct 30 06:15:03.454247 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.454229    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" event={"ID":"404e0b49-889c-4102-91fd-b989dcd5445c","Type":"ContainerDied","Data":"b84277bf73d0c709251d57c6a35f17c4ff9ff073ef7874be95c3ae400a70ba18"}
Oct 30 06:15:03.454670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.454652    5250 scope.go:122] "RemoveContainer" containerID="b84277bf73d0c709251d57c6a35f17c4ff9ff073ef7874be95c3ae400a70ba18"
Oct 30 06:15:03.487522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.486976    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2" event={"ID":"4a0f257f-bf55-40dc-81b1-af99c919362a","Type":"ContainerDied","Data":"0548e7ed4351df0e8944435355fa113f8ca244385aa4447b8532c152b073081e"}
Oct 30 06:15:03.487522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.487031    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0548e7ed4351df0e8944435355fa113f8ca244385aa4447b8532c152b073081e"
Oct 30 06:15:03.487522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.487283    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"
Oct 30 06:15:03.493137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.493085    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" podStartSLOduration=3.493069472 podStartE2EDuration="3.493069472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.482244984 +0000 UTC m=+2618.313282216" watchObservedRunningTime="2026-10-30 06:15:03.493069472 +0000 UTC m=+2618.324106693"
Oct 30 06:15:03.502473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.502447    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" event={"ID":"259369fd-d911-4b38-96d2-ee3b43d3c075","Type":"ContainerStarted","Data":"8fba8868b7957992514e66dbd63fbea3bc22dac3abca3823d94f61e894ce8e43"}
Oct 30 06:15:03.529539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.526060    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" podStartSLOduration=3.526043983 podStartE2EDuration="3.526043983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.520714316 +0000 UTC m=+2618.351751532" watchObservedRunningTime="2026-10-30 06:15:03.526043983 +0000 UTC m=+2618.357081201"
Oct 30 06:15:03.530167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.530137    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" event={"ID":"c9c63877-7c54-427d-b8bf-c0f1b22dcebb","Type":"ContainerStarted","Data":"cde873771a4d073d19ff61065d70cd66d9fdd00eb218faef0d548298b0a88511"}
Oct 30 06:15:03.530962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.530935    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" event={"ID":"bc73291b-82f1-4261-bc0b-8712d8a49558","Type":"ContainerStarted","Data":"6315cee8b224bd4ee89a2ea5e55c090ae044f5cc843d2254bae7d062e8864529"}
Oct 30 06:15:03.533329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.533305    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" event={"ID":"d09445ae-5f09-4c18-a37e-3af42038cd1e","Type":"ContainerStarted","Data":"e2cdbc57b6b599ab1b33d7e91359086fd9b5a0e13d3ff77af89e44b89d05d649"}
Oct 30 06:15:03.535539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.535519    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" event={"ID":"ab62622d-84ad-4903-9be4-3b6ec51291f3","Type":"ContainerStarted","Data":"11584ac6750ac2930d3e11bc307db4e739ae405af5223baf887e09b2eae5d159"}
Oct 30 06:15:03.537349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.537326    5250 generic.go:356] "Generic (PLEG): container finished" podID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerID="09ce84b2840bd0142299f8ad36d1c16525be4cc2ea428dba940c569c5c165c88" exitCode=60
Oct 30 06:15:03.537456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.537409    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" event={"ID":"f3be9b16-06d1-4ab5-a8e8-1825afb98f39","Type":"ContainerDied","Data":"09ce84b2840bd0142299f8ad36d1c16525be4cc2ea428dba940c569c5c165c88"}
Oct 30 06:15:03.537602 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.537588    5250 scope.go:122] "RemoveContainer" containerID="09ce84b2840bd0142299f8ad36d1c16525be4cc2ea428dba940c569c5c165c88"
Oct 30 06:15:03.566753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.566700    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" podStartSLOduration=3.566680524 podStartE2EDuration="3.566680524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.56653285 +0000 UTC m=+2618.397570070" watchObservedRunningTime="2026-10-30 06:15:03.566680524 +0000 UTC m=+2618.397717742"
Oct 30 06:15:03.607359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.606108    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" podStartSLOduration=3.606091245 podStartE2EDuration="3.606091245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.602425219 +0000 UTC m=+2618.433462438" watchObservedRunningTime="2026-10-30 06:15:03.606091245 +0000 UTC m=+2618.437128465"
Oct 30 06:15:03.716294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.716223    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" podStartSLOduration=3.7162048 podStartE2EDuration="3.7162048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.715625156 +0000 UTC m=+2618.546662385" watchObservedRunningTime="2026-10-30 06:15:03.7162048 +0000 UTC m=+2618.547242017"
Oct 30 06:15:03.756712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.756664    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" podStartSLOduration=3.756650762 podStartE2EDuration="3.756650762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.75101437 +0000 UTC m=+2618.582051588" watchObservedRunningTime="2026-10-30 06:15:03.756650762 +0000 UTC m=+2618.587687978"
Oct 30 06:15:03.826433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.826374    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" podStartSLOduration=3.826355612 podStartE2EDuration="3.826355612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.824783427 +0000 UTC m=+2618.655820644" watchObservedRunningTime="2026-10-30 06:15:03.826355612 +0000 UTC m=+2618.657392828"
Oct 30 06:15:03.912172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.912109    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" podStartSLOduration=3.912090202 podStartE2EDuration="3.912090202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.871427882 +0000 UTC m=+2618.702465099" watchObservedRunningTime="2026-10-30 06:15:03.912090202 +0000 UTC m=+2618.743127421"
Oct 30 06:15:03.912376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:03.912252    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" podStartSLOduration=3.9122439890000003 podStartE2EDuration="3.912243989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.907045233 +0000 UTC m=+2618.738082449" watchObservedRunningTime="2026-10-30 06:15:03.912243989 +0000 UTC m=+2618.743281211"
Oct 30 06:15:04.168060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.168038    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:15:04.242046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242010    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-token\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-token\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242076    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-podinfo\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242137    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-envoy\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242375 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242355    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-socket\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242411    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-data\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242517    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-kube-api-access-nzprr\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-kube-api-access-nzprr\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242569    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-credential-socket\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242610    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-certs\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242683    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istiod-ca-cert\") pod \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\" (UID: \"9ba977e9-e030-40a0-a519-9b3fc81b1a16\") "
Oct 30 06:15:04.242939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.242911    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-socket" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:04.243654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.243571    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-data" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:04.243763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.243658    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-certs" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:04.247223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.244372    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-credential-socket" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:04.247223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.245494    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istiod-ca-cert" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:04.247223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.246361    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-kube-api-access-nzprr" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "kube-api-access-nzprr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:04.247223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.246860    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-token" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:04.247223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.246872    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-podinfo" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:15:04.248828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.248803    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-envoy" pod "9ba977e9-e030-40a0-a519-9b3fc81b1a16" (UID: "9ba977e9-e030-40a0-a519-9b3fc81b1a16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:04.344801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344764    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.344801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344794    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.344801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344804    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzprr\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-kube-api-access-nzprr\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.344801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344813    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.345097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344824    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.345097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344836    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.345097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344845    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.345097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344852    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.345097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.344859    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba977e9-e030-40a0-a519-9b3fc81b1a16-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:04.548879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.548784    5250 generic.go:356] "Generic (PLEG): container finished" podID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerID="cde873771a4d073d19ff61065d70cd66d9fdd00eb218faef0d548298b0a88511" exitCode=60
Oct 30 06:15:04.549363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.548873    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" event={"ID":"c9c63877-7c54-427d-b8bf-c0f1b22dcebb","Type":"ContainerDied","Data":"cde873771a4d073d19ff61065d70cd66d9fdd00eb218faef0d548298b0a88511"}
Oct 30 06:15:04.549363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.549167    5250 scope.go:122] "RemoveContainer" containerID="cde873771a4d073d19ff61065d70cd66d9fdd00eb218faef0d548298b0a88511"
Oct 30 06:15:04.550707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.550684    5250 generic.go:356] "Generic (PLEG): container finished" podID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerID="6315cee8b224bd4ee89a2ea5e55c090ae044f5cc843d2254bae7d062e8864529" exitCode=60
Oct 30 06:15:04.550828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.550772    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" event={"ID":"bc73291b-82f1-4261-bc0b-8712d8a49558","Type":"ContainerDied","Data":"6315cee8b224bd4ee89a2ea5e55c090ae044f5cc843d2254bae7d062e8864529"}
Oct 30 06:15:04.551075 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.551006    5250 scope.go:122] "RemoveContainer" containerID="6315cee8b224bd4ee89a2ea5e55c090ae044f5cc843d2254bae7d062e8864529"
Oct 30 06:15:04.552787 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.552728    5250 generic.go:356] "Generic (PLEG): container finished" podID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerID="e2cdbc57b6b599ab1b33d7e91359086fd9b5a0e13d3ff77af89e44b89d05d649" exitCode=60
Oct 30 06:15:04.552848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.552783    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" event={"ID":"d09445ae-5f09-4c18-a37e-3af42038cd1e","Type":"ContainerDied","Data":"e2cdbc57b6b599ab1b33d7e91359086fd9b5a0e13d3ff77af89e44b89d05d649"}
Oct 30 06:15:04.553089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.553072    5250 scope.go:122] "RemoveContainer" containerID="e2cdbc57b6b599ab1b33d7e91359086fd9b5a0e13d3ff77af89e44b89d05d649"
Oct 30 06:15:04.555129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.555108    5250 generic.go:356] "Generic (PLEG): container finished" podID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerID="11584ac6750ac2930d3e11bc307db4e739ae405af5223baf887e09b2eae5d159" exitCode=60
Oct 30 06:15:04.555269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.555163    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" event={"ID":"ab62622d-84ad-4903-9be4-3b6ec51291f3","Type":"ContainerDied","Data":"11584ac6750ac2930d3e11bc307db4e739ae405af5223baf887e09b2eae5d159"}
Oct 30 06:15:04.555419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.555403    5250 scope.go:122] "RemoveContainer" containerID="11584ac6750ac2930d3e11bc307db4e739ae405af5223baf887e09b2eae5d159"
Oct 30 06:15:04.557708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.557685    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" event={"ID":"f3be9b16-06d1-4ab5-a8e8-1825afb98f39","Type":"ContainerStarted","Data":"6ffd0f2d37f158fb53049b3fc5cd82a9ddfd147e809e445a2e6de04e9edff92d"}
Oct 30 06:15:04.560330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.560308    5250 generic.go:356] "Generic (PLEG): container finished" podID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerID="40b4d006c0c48e86ba8d1508df9b6652e9346b161878c8e0e813f83520f719bf" exitCode=60
Oct 30 06:15:04.560450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.560392    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" event={"ID":"27ebfb55-80c6-4651-b46c-e588b914c30c","Type":"ContainerDied","Data":"40b4d006c0c48e86ba8d1508df9b6652e9346b161878c8e0e813f83520f719bf"}
Oct 30 06:15:04.560595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.560578    5250 scope.go:122] "RemoveContainer" containerID="40b4d006c0c48e86ba8d1508df9b6652e9346b161878c8e0e813f83520f719bf"
Oct 30 06:15:04.563483 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.563464    5250 generic.go:356] "Generic (PLEG): container finished" podID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerID="7dee05fb3f20f56a8d8e933d7f2c24e6c4231cc14e2070726abdb059fae74ac1" exitCode=60
Oct 30 06:15:04.563574 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.563526    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" event={"ID":"d01694ac-ebec-4c70-a10a-39f4de159d25","Type":"ContainerDied","Data":"7dee05fb3f20f56a8d8e933d7f2c24e6c4231cc14e2070726abdb059fae74ac1"}
Oct 30 06:15:04.563856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.563837    5250 scope.go:122] "RemoveContainer" containerID="7dee05fb3f20f56a8d8e933d7f2c24e6c4231cc14e2070726abdb059fae74ac1"
Oct 30 06:15:04.566461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.566442    5250 generic.go:356] "Generic (PLEG): container finished" podID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerID="4574b1bea32b038e506c864b0bedefd10f272b91d1d1751f9af45c43f8b9ed05" exitCode=60
Oct 30 06:15:04.566898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.566878    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" event={"ID":"8e3d2145-c914-4668-b07d-c7058b2d3f59","Type":"ContainerDied","Data":"4574b1bea32b038e506c864b0bedefd10f272b91d1d1751f9af45c43f8b9ed05"}
Oct 30 06:15:04.568011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.567832    5250 scope.go:122] "RemoveContainer" containerID="4574b1bea32b038e506c864b0bedefd10f272b91d1d1751f9af45c43f8b9ed05"
Oct 30 06:15:04.574388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.574364    5250 generic.go:356] "Generic (PLEG): container finished" podID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerID="ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501" exitCode=0
Oct 30 06:15:04.574497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.574472    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" event={"ID":"9ba977e9-e030-40a0-a519-9b3fc81b1a16","Type":"ContainerDied","Data":"ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501"}
Oct 30 06:15:04.574564 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.574519    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp" event={"ID":"9ba977e9-e030-40a0-a519-9b3fc81b1a16","Type":"ContainerDied","Data":"c48783607ef5a748b68b9223fa666e90eaa9a2d289934c745644a5f278b5a77a"}
Oct 30 06:15:04.574564 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.574548    5250 scope.go:122] "RemoveContainer" containerID="ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501"
Oct 30 06:15:04.574720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.574705    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"
Oct 30 06:15:04.581290 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.581264    5250 generic.go:356] "Generic (PLEG): container finished" podID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerID="b20934a2c31aa2c8c4399f0a4af8e58d3fe61ec32852b5a785481642416059a8" exitCode=60
Oct 30 06:15:04.581406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.581344    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" event={"ID":"4c5783c3-b770-4ded-a301-7e607e09bdb4","Type":"ContainerDied","Data":"b20934a2c31aa2c8c4399f0a4af8e58d3fe61ec32852b5a785481642416059a8"}
Oct 30 06:15:04.581759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.581739    5250 scope.go:122] "RemoveContainer" containerID="b20934a2c31aa2c8c4399f0a4af8e58d3fe61ec32852b5a785481642416059a8"
Oct 30 06:15:04.588144 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.588116    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" event={"ID":"6fd00ce8-6874-4369-9022-1d1001fcbae3","Type":"ContainerStarted","Data":"65e17cfcf029f065be7b15499e759b790be441738eb9ac7c9629378156bcfd5a"}
Oct 30 06:15:04.592334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.592029    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" event={"ID":"404e0b49-889c-4102-91fd-b989dcd5445c","Type":"ContainerStarted","Data":"8de496c4884d46c24697e89dc4368a2f86026d4e7a1a1e8df0d849a48df1f89f"}
Oct 30 06:15:04.600753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.600729    5250 generic.go:356] "Generic (PLEG): container finished" podID="ff353edf-ad97-4352-a8b2-470fa965104c" containerID="78f9f990301e413b26aed86d3dc32e0e93aa279e95a469316dc87e7c1b97164b" exitCode=60
Oct 30 06:15:04.600957 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.600811    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" event={"ID":"ff353edf-ad97-4352-a8b2-470fa965104c","Type":"ContainerDied","Data":"78f9f990301e413b26aed86d3dc32e0e93aa279e95a469316dc87e7c1b97164b"}
Oct 30 06:15:04.601235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.601211    5250 scope.go:122] "RemoveContainer" containerID="78f9f990301e413b26aed86d3dc32e0e93aa279e95a469316dc87e7c1b97164b"
Oct 30 06:15:04.626829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.625606    5250 scope.go:122] "RemoveContainer" containerID="ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501"
Oct 30 06:15:04.628105 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:04.628075    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501\": container with ID starting with ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501 not found: ID does not exist" containerID="ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501"
Oct 30 06:15:04.628241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.628118    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501"} err="failed to get container status \"ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501\": rpc error: code = NotFound desc = could not find container \"ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501\": container with ID starting with ca401ac26d0b664e1fd918532a590b7688f867826ec9f5d6f768c3e629e99501 not found: ID does not exist"
Oct 30 06:15:04.628916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.628844    5250 generic.go:356] "Generic (PLEG): container finished" podID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerID="8fba8868b7957992514e66dbd63fbea3bc22dac3abca3823d94f61e894ce8e43" exitCode=60
Oct 30 06:15:04.628916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.628879    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" event={"ID":"259369fd-d911-4b38-96d2-ee3b43d3c075","Type":"ContainerDied","Data":"8fba8868b7957992514e66dbd63fbea3bc22dac3abca3823d94f61e894ce8e43"}
Oct 30 06:15:04.629266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.629245    5250 scope.go:122] "RemoveContainer" containerID="8fba8868b7957992514e66dbd63fbea3bc22dac3abca3823d94f61e894ce8e43"
Oct 30 06:15:04.825685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.825598    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:04.978960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:04.978914    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" podStartSLOduration=4.978900176 podStartE2EDuration="4.978900176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.973774134 +0000 UTC m=+2619.804811350" watchObservedRunningTime="2026-10-30 06:15:04.978900176 +0000 UTC m=+2619.809937434"
Oct 30 06:15:05.032715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.023707    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:05.138208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.134248    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"]
Oct 30 06:15:05.147214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.146594    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-tmbhp"]
Oct 30 06:15:05.288171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.288138    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" path="/var/lib/kubelet/pods/9ba977e9-e030-40a0-a519-9b3fc81b1a16/volumes"
Oct 30 06:15:05.642258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.642223    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" event={"ID":"27ebfb55-80c6-4651-b46c-e588b914c30c","Type":"ContainerStarted","Data":"ecb2d0486d2892b23c5453669b921ad37327976912541aa4a8810ca53283fa66"}
Oct 30 06:15:05.644474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.644445    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" event={"ID":"d01694ac-ebec-4c70-a10a-39f4de159d25","Type":"ContainerStarted","Data":"db84471f566699317752eae6c6b340ce2acba9427a92287867f36742cd4e900d"}
Oct 30 06:15:05.646692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.646649    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" event={"ID":"8e3d2145-c914-4668-b07d-c7058b2d3f59","Type":"ContainerStarted","Data":"00fc60f42e6268bca7d5fb01573878c193329d1d9db6b6bc70bc63d960ad98ed"}
Oct 30 06:15:05.649716 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.649693    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" event={"ID":"4c5783c3-b770-4ded-a301-7e607e09bdb4","Type":"ContainerStarted","Data":"68c18e3345422b71c040dff56d7515374655b1851268eea9ea24d5f643eb535b"}
Oct 30 06:15:05.651590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.651568    5250 generic.go:356] "Generic (PLEG): container finished" podID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerID="65e17cfcf029f065be7b15499e759b790be441738eb9ac7c9629378156bcfd5a" exitCode=60
Oct 30 06:15:05.651713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.651648    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" event={"ID":"6fd00ce8-6874-4369-9022-1d1001fcbae3","Type":"ContainerDied","Data":"65e17cfcf029f065be7b15499e759b790be441738eb9ac7c9629378156bcfd5a"}
Oct 30 06:15:05.651791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.651714    5250 scope.go:122] "RemoveContainer" containerID="e7217eee0e87cb5fd198edaa930dd800edd95e86a62a3e7ebd84a7bc334e79cb"
Oct 30 06:15:05.651885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.651871    5250 scope.go:122] "RemoveContainer" containerID="65e17cfcf029f065be7b15499e759b790be441738eb9ac7c9629378156bcfd5a"
Oct 30 06:15:05.652214 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:05.652170    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b_odh-ai-gateway-infra(6fd00ce8-6874-4369-9022-1d1001fcbae3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3"
Oct 30 06:15:05.653805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.653784    5250 generic.go:356] "Generic (PLEG): container finished" podID="404e0b49-889c-4102-91fd-b989dcd5445c" containerID="8de496c4884d46c24697e89dc4368a2f86026d4e7a1a1e8df0d849a48df1f89f" exitCode=60
Oct 30 06:15:05.653889 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.653820    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" event={"ID":"404e0b49-889c-4102-91fd-b989dcd5445c","Type":"ContainerDied","Data":"8de496c4884d46c24697e89dc4368a2f86026d4e7a1a1e8df0d849a48df1f89f"}
Oct 30 06:15:05.654066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.654051    5250 scope.go:122] "RemoveContainer" containerID="8de496c4884d46c24697e89dc4368a2f86026d4e7a1a1e8df0d849a48df1f89f"
Oct 30 06:15:05.654364 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:05.654338    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw_odh-ai-gateway-infra(404e0b49-889c-4102-91fd-b989dcd5445c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" podUID="404e0b49-889c-4102-91fd-b989dcd5445c"
Oct 30 06:15:05.661633 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.661608    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" event={"ID":"ff353edf-ad97-4352-a8b2-470fa965104c","Type":"ContainerStarted","Data":"e7e9a3d9eb4333bd5b2f832d708cce50d5770d4829202631c95dd52a7836599e"}
Oct 30 06:15:05.667150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.667121    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" event={"ID":"259369fd-d911-4b38-96d2-ee3b43d3c075","Type":"ContainerStarted","Data":"be5fcb5c1e1eea007e5dac986e235aef819b5ad96e2cbbfe848515b01dc82ab6"}
Oct 30 06:15:05.669585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.669557    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" event={"ID":"c9c63877-7c54-427d-b8bf-c0f1b22dcebb","Type":"ContainerStarted","Data":"1cd30f877aa59eb96a8e8ab7c7b5fd7427bdebe6790615d6675861a65710eefe"}
Oct 30 06:15:05.671604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.671580    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" event={"ID":"bc73291b-82f1-4261-bc0b-8712d8a49558","Type":"ContainerStarted","Data":"e80ab7cc63315f08e7a8b60f9ebc7ba035b12837a4d3f9a6d6b2c4d525d0d366"}
Oct 30 06:15:05.672813 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.672798    5250 scope.go:122] "RemoveContainer" containerID="b84277bf73d0c709251d57c6a35f17c4ff9ff073ef7874be95c3ae400a70ba18"
Oct 30 06:15:05.673601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.673579    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" event={"ID":"d09445ae-5f09-4c18-a37e-3af42038cd1e","Type":"ContainerStarted","Data":"ea68c3d10b3ec16bd5d61b8d6bbfeb3272a7edc9e1d1f6fef59ba6e8e74ee9f4"}
Oct 30 06:15:05.675698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.675675    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" event={"ID":"ab62622d-84ad-4903-9be4-3b6ec51291f3","Type":"ContainerStarted","Data":"e36f8bf952eb1fe0395b43013d717fd0f0cc942cf19680b1926901dd9bef0f46"}
Oct 30 06:15:05.677748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.677728    5250 generic.go:356] "Generic (PLEG): container finished" podID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerID="6ffd0f2d37f158fb53049b3fc5cd82a9ddfd147e809e445a2e6de04e9edff92d" exitCode=60
Oct 30 06:15:05.677830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.677777    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" event={"ID":"f3be9b16-06d1-4ab5-a8e8-1825afb98f39","Type":"ContainerDied","Data":"6ffd0f2d37f158fb53049b3fc5cd82a9ddfd147e809e445a2e6de04e9edff92d"}
Oct 30 06:15:05.678369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.678067    5250 scope.go:122] "RemoveContainer" containerID="6ffd0f2d37f158fb53049b3fc5cd82a9ddfd147e809e445a2e6de04e9edff92d"
Oct 30 06:15:05.678468 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:05.678385    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb_odh-ai-gateway-infra(f3be9b16-06d1-4ab5-a8e8-1825afb98f39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39"
Oct 30 06:15:05.688995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.688974    5250 scope.go:122] "RemoveContainer" containerID="09ce84b2840bd0142299f8ad36d1c16525be4cc2ea428dba940c569c5c165c88"
Oct 30 06:15:05.719582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:05.719534    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" podStartSLOduration=5.719522162 podStartE2EDuration="5.719522162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:05.715225167 +0000 UTC m=+2620.546262378" watchObservedRunningTime="2026-10-30 06:15:05.719522162 +0000 UTC m=+2620.550559378"
Oct 30 06:15:06.518228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.518088    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:06.588300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.588256    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"]
Oct 30 06:15:06.589143 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.589123    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy"
Oct 30 06:15:06.589270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.589145    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy"
Oct 30 06:15:06.589270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.589161    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4a0f257f-bf55-40dc-81b1-af99c919362a" containerName="revoke-keys"
Oct 30 06:15:06.589270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.589166    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0f257f-bf55-40dc-81b1-af99c919362a" containerName="revoke-keys"
Oct 30 06:15:06.589377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.589366    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba977e9-e030-40a0-a519-9b3fc81b1a16" containerName="istio-proxy"
Oct 30 06:15:06.589416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.589379    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0f257f-bf55-40dc-81b1-af99c919362a" containerName="revoke-keys"
Oct 30 06:15:06.593545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.593521    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:06.615679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.615645    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-de2bf2\""
Oct 30 06:15:06.615805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.615748    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-de2bf2-dockercfg-nkbp2\""
Oct 30 06:15:06.624081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.624054    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"]
Oct 30 06:15:06.689795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.689761    5250 generic.go:356] "Generic (PLEG): container finished" podID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerID="e80ab7cc63315f08e7a8b60f9ebc7ba035b12837a4d3f9a6d6b2c4d525d0d366" exitCode=60
Oct 30 06:15:06.690339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.689849    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" event={"ID":"bc73291b-82f1-4261-bc0b-8712d8a49558","Type":"ContainerDied","Data":"e80ab7cc63315f08e7a8b60f9ebc7ba035b12837a4d3f9a6d6b2c4d525d0d366"}
Oct 30 06:15:06.690339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.689931    5250 scope.go:122] "RemoveContainer" containerID="6315cee8b224bd4ee89a2ea5e55c090ae044f5cc843d2254bae7d062e8864529"
Oct 30 06:15:06.690339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.690279    5250 scope.go:122] "RemoveContainer" containerID="e80ab7cc63315f08e7a8b60f9ebc7ba035b12837a4d3f9a6d6b2c4d525d0d366"
Oct 30 06:15:06.690720 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.690698    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p_odh-ai-gateway-infra(bc73291b-82f1-4261-bc0b-8712d8a49558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558"
Oct 30 06:15:06.692241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.692218    5250 generic.go:356] "Generic (PLEG): container finished" podID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerID="ea68c3d10b3ec16bd5d61b8d6bbfeb3272a7edc9e1d1f6fef59ba6e8e74ee9f4" exitCode=60
Oct 30 06:15:06.692349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.692279    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" event={"ID":"d09445ae-5f09-4c18-a37e-3af42038cd1e","Type":"ContainerDied","Data":"ea68c3d10b3ec16bd5d61b8d6bbfeb3272a7edc9e1d1f6fef59ba6e8e74ee9f4"}
Oct 30 06:15:06.692573 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.692560    5250 scope.go:122] "RemoveContainer" containerID="ea68c3d10b3ec16bd5d61b8d6bbfeb3272a7edc9e1d1f6fef59ba6e8e74ee9f4"
Oct 30 06:15:06.692938 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.692916    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9_odh-ai-gateway-infra(d09445ae-5f09-4c18-a37e-3af42038cd1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e"
Oct 30 06:15:06.694575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.694557    5250 generic.go:356] "Generic (PLEG): container finished" podID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerID="e36f8bf952eb1fe0395b43013d717fd0f0cc942cf19680b1926901dd9bef0f46" exitCode=60
Oct 30 06:15:06.694665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.694602    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" event={"ID":"ab62622d-84ad-4903-9be4-3b6ec51291f3","Type":"ContainerDied","Data":"e36f8bf952eb1fe0395b43013d717fd0f0cc942cf19680b1926901dd9bef0f46"}
Oct 30 06:15:06.694942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.694926    5250 scope.go:122] "RemoveContainer" containerID="e36f8bf952eb1fe0395b43013d717fd0f0cc942cf19680b1926901dd9bef0f46"
Oct 30 06:15:06.695326 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.695303    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-rrn6s_odh-ai-gateway-infra(ab62622d-84ad-4903-9be4-3b6ec51291f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3"
Oct 30 06:15:06.695578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.695557    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8009548a-1020-4ebb-bb28-a363a1b6f9ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f\" (UID: \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:06.695876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.695853    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gflc5\" (UniqueName: \"kubernetes.io/projected/8009548a-1020-4ebb-bb28-a363a1b6f9ee-kube-api-access-gflc5\") pod \"payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f\" (UID: \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:06.696836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.696817    5250 scope.go:122] "RemoveContainer" containerID="6ffd0f2d37f158fb53049b3fc5cd82a9ddfd147e809e445a2e6de04e9edff92d"
Oct 30 06:15:06.697065 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.697042    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb_odh-ai-gateway-infra(f3be9b16-06d1-4ab5-a8e8-1825afb98f39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39"
Oct 30 06:15:06.698606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.698586    5250 generic.go:356] "Generic (PLEG): container finished" podID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerID="ecb2d0486d2892b23c5453669b921ad37327976912541aa4a8810ca53283fa66" exitCode=60
Oct 30 06:15:06.698700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.698653    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" event={"ID":"27ebfb55-80c6-4651-b46c-e588b914c30c","Type":"ContainerDied","Data":"ecb2d0486d2892b23c5453669b921ad37327976912541aa4a8810ca53283fa66"}
Oct 30 06:15:06.698904 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.698889    5250 scope.go:122] "RemoveContainer" containerID="ecb2d0486d2892b23c5453669b921ad37327976912541aa4a8810ca53283fa66"
Oct 30 06:15:06.699208 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.699145    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px_odh-ai-gateway-infra(27ebfb55-80c6-4651-b46c-e588b914c30c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c"
Oct 30 06:15:06.700626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.700597    5250 generic.go:356] "Generic (PLEG): container finished" podID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerID="db84471f566699317752eae6c6b340ce2acba9427a92287867f36742cd4e900d" exitCode=60
Oct 30 06:15:06.700697 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.700670    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" event={"ID":"d01694ac-ebec-4c70-a10a-39f4de159d25","Type":"ContainerDied","Data":"db84471f566699317752eae6c6b340ce2acba9427a92287867f36742cd4e900d"}
Oct 30 06:15:06.700884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.700869    5250 scope.go:122] "RemoveContainer" containerID="db84471f566699317752eae6c6b340ce2acba9427a92287867f36742cd4e900d"
Oct 30 06:15:06.701121 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.701101    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9_odh-ai-gateway-infra(d01694ac-ebec-4c70-a10a-39f4de159d25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25"
Oct 30 06:15:06.702829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.702804    5250 generic.go:356] "Generic (PLEG): container finished" podID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerID="00fc60f42e6268bca7d5fb01573878c193329d1d9db6b6bc70bc63d960ad98ed" exitCode=60
Oct 30 06:15:06.702911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.702848    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" event={"ID":"8e3d2145-c914-4668-b07d-c7058b2d3f59","Type":"ContainerDied","Data":"00fc60f42e6268bca7d5fb01573878c193329d1d9db6b6bc70bc63d960ad98ed"}
Oct 30 06:15:06.703150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.703132    5250 scope.go:122] "RemoveContainer" containerID="00fc60f42e6268bca7d5fb01573878c193329d1d9db6b6bc70bc63d960ad98ed"
Oct 30 06:15:06.703516 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.703461    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h_odh-ai-gateway-infra(8e3d2145-c914-4668-b07d-c7058b2d3f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59"
Oct 30 06:15:06.705694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.705666    5250 generic.go:356] "Generic (PLEG): container finished" podID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerID="68c18e3345422b71c040dff56d7515374655b1851268eea9ea24d5f643eb535b" exitCode=60
Oct 30 06:15:06.705835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.705749    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" event={"ID":"4c5783c3-b770-4ded-a301-7e607e09bdb4","Type":"ContainerDied","Data":"68c18e3345422b71c040dff56d7515374655b1851268eea9ea24d5f643eb535b"}
Oct 30 06:15:06.706276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.706251    5250 scope.go:122] "RemoveContainer" containerID="68c18e3345422b71c040dff56d7515374655b1851268eea9ea24d5f643eb535b"
Oct 30 06:15:06.706594 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.706571    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq_odh-ai-gateway-infra(4c5783c3-b770-4ded-a301-7e607e09bdb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4"
Oct 30 06:15:06.709329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.708214    5250 scope.go:122] "RemoveContainer" containerID="e2cdbc57b6b599ab1b33d7e91359086fd9b5a0e13d3ff77af89e44b89d05d649"
Oct 30 06:15:06.841794 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.820988    5250 scope.go:122] "RemoveContainer" containerID="65e17cfcf029f065be7b15499e759b790be441738eb9ac7c9629378156bcfd5a"
Oct 30 06:15:06.841794 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.821471    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b_odh-ai-gateway-infra(6fd00ce8-6874-4369-9022-1d1001fcbae3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3"
Oct 30 06:15:06.877905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.877855    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:06.881814 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.881783    5250 scope.go:122] "RemoveContainer" containerID="8de496c4884d46c24697e89dc4368a2f86026d4e7a1a1e8df0d849a48df1f89f"
Oct 30 06:15:06.882945 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.882922    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw_odh-ai-gateway-infra(404e0b49-889c-4102-91fd-b989dcd5445c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" podUID="404e0b49-889c-4102-91fd-b989dcd5445c"
Oct 30 06:15:06.905903 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.905785    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8009548a-1020-4ebb-bb28-a363a1b6f9ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f\" (UID: \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:06.906445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.906416    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflc5\" (UniqueName: \"kubernetes.io/projected/8009548a-1020-4ebb-bb28-a363a1b6f9ee-kube-api-access-gflc5\") pod \"payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f\" (UID: \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:06.908317 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.908283    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8009548a-1020-4ebb-bb28-a363a1b6f9ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f\" (UID: \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:06.933353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.932678    5250 scope.go:122] "RemoveContainer" containerID="11584ac6750ac2930d3e11bc307db4e739ae405af5223baf887e09b2eae5d159"
Oct 30 06:15:06.933353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.932935    5250 generic.go:356] "Generic (PLEG): container finished" podID="ff353edf-ad97-4352-a8b2-470fa965104c" containerID="e7e9a3d9eb4333bd5b2f832d708cce50d5770d4829202631c95dd52a7836599e" exitCode=60
Oct 30 06:15:06.933353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.933018    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" event={"ID":"ff353edf-ad97-4352-a8b2-470fa965104c","Type":"ContainerDied","Data":"e7e9a3d9eb4333bd5b2f832d708cce50d5770d4829202631c95dd52a7836599e"}
Oct 30 06:15:06.938691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.938670    5250 scope.go:122] "RemoveContainer" containerID="e7e9a3d9eb4333bd5b2f832d708cce50d5770d4829202631c95dd52a7836599e"
Oct 30 06:15:06.943736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.943704    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gflc5\" (UniqueName: \"kubernetes.io/projected/8009548a-1020-4ebb-bb28-a363a1b6f9ee-kube-api-access-gflc5\") pod \"payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f\" (UID: \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:06.944550 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.944520    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6_odh-ai-gateway-infra(ff353edf-ad97-4352-a8b2-470fa965104c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" podUID="ff353edf-ad97-4352-a8b2-470fa965104c"
Oct 30 06:15:06.959468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.958860    5250 generic.go:356] "Generic (PLEG): container finished" podID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerID="be5fcb5c1e1eea007e5dac986e235aef819b5ad96e2cbbfe848515b01dc82ab6" exitCode=60
Oct 30 06:15:06.959468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.958999    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" event={"ID":"259369fd-d911-4b38-96d2-ee3b43d3c075","Type":"ContainerDied","Data":"be5fcb5c1e1eea007e5dac986e235aef819b5ad96e2cbbfe848515b01dc82ab6"}
Oct 30 06:15:06.960223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.959901    5250 scope.go:122] "RemoveContainer" containerID="be5fcb5c1e1eea007e5dac986e235aef819b5ad96e2cbbfe848515b01dc82ab6"
Oct 30 06:15:06.960378 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.960355    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2_odh-ai-gateway-infra(259369fd-d911-4b38-96d2-ee3b43d3c075)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075"
Oct 30 06:15:06.961344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.961316    5250 generic.go:356] "Generic (PLEG): container finished" podID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerID="1cd30f877aa59eb96a8e8ab7c7b5fd7427bdebe6790615d6675861a65710eefe" exitCode=60
Oct 30 06:15:06.961429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.961382    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" event={"ID":"c9c63877-7c54-427d-b8bf-c0f1b22dcebb","Type":"ContainerDied","Data":"1cd30f877aa59eb96a8e8ab7c7b5fd7427bdebe6790615d6675861a65710eefe"}
Oct 30 06:15:06.961780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.961760    5250 scope.go:122] "RemoveContainer" containerID="1cd30f877aa59eb96a8e8ab7c7b5fd7427bdebe6790615d6675861a65710eefe"
Oct 30 06:15:06.962229 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:06.962201    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9_odh-ai-gateway-infra(c9c63877-7c54-427d-b8bf-c0f1b22dcebb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb"
Oct 30 06:15:06.973053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.973024    5250 scope.go:122] "RemoveContainer" containerID="40b4d006c0c48e86ba8d1508df9b6652e9346b161878c8e0e813f83520f719bf"
Oct 30 06:15:06.992729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:06.992699    5250 scope.go:122] "RemoveContainer" containerID="7dee05fb3f20f56a8d8e933d7f2c24e6c4231cc14e2070726abdb059fae74ac1"
Oct 30 06:15:07.007960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.007939    5250 scope.go:122] "RemoveContainer" containerID="4574b1bea32b038e506c864b0bedefd10f272b91d1d1751f9af45c43f8b9ed05"
Oct 30 06:15:07.023316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.023277    5250 scope.go:122] "RemoveContainer" containerID="b20934a2c31aa2c8c4399f0a4af8e58d3fe61ec32852b5a785481642416059a8"
Oct 30 06:15:07.038759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.038736    5250 scope.go:122] "RemoveContainer" containerID="78f9f990301e413b26aed86d3dc32e0e93aa279e95a469316dc87e7c1b97164b"
Oct 30 06:15:07.054944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.054923    5250 scope.go:122] "RemoveContainer" containerID="8fba8868b7957992514e66dbd63fbea3bc22dac3abca3823d94f61e894ce8e43"
Oct 30 06:15:07.073625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.072586    5250 scope.go:122] "RemoveContainer" containerID="cde873771a4d073d19ff61065d70cd66d9fdd00eb218faef0d548298b0a88511"
Oct 30 06:15:07.118907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.118880    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:15:07.119252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.119230    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:15:07.208514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.208482    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:07.259594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.259555    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:07.548540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.548509    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"]
Oct 30 06:15:07.548898 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:07.548875    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8009548a_1020_4ebb_bb28_a363a1b6f9ee.slice/crio-22356e674f1cd400836617464aa629bca3f5b253dd5cdf4474636eee090b332c WatchSource:0}: Error finding container 22356e674f1cd400836617464aa629bca3f5b253dd5cdf4474636eee090b332c: Status 404 returned error can't find the container with id 22356e674f1cd400836617464aa629bca3f5b253dd5cdf4474636eee090b332c
Oct 30 06:15:07.974639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.974609    5250 scope.go:122] "RemoveContainer" containerID="ecb2d0486d2892b23c5453669b921ad37327976912541aa4a8810ca53283fa66"
Oct 30 06:15:07.975129 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.975029    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px_odh-ai-gateway-infra(27ebfb55-80c6-4651-b46c-e588b914c30c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c"
Oct 30 06:15:07.976448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.976423    5250 scope.go:122] "RemoveContainer" containerID="db84471f566699317752eae6c6b340ce2acba9427a92287867f36742cd4e900d"
Oct 30 06:15:07.976798 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.976771    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9_odh-ai-gateway-infra(d01694ac-ebec-4c70-a10a-39f4de159d25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25"
Oct 30 06:15:07.978117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.978099    5250 scope.go:122] "RemoveContainer" containerID="00fc60f42e6268bca7d5fb01573878c193329d1d9db6b6bc70bc63d960ad98ed"
Oct 30 06:15:07.978459 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.978442    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h_odh-ai-gateway-infra(8e3d2145-c914-4668-b07d-c7058b2d3f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59"
Oct 30 06:15:07.979762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.979740    5250 scope.go:122] "RemoveContainer" containerID="68c18e3345422b71c040dff56d7515374655b1851268eea9ea24d5f643eb535b"
Oct 30 06:15:07.979988 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.979970    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq_odh-ai-gateway-infra(4c5783c3-b770-4ded-a301-7e607e09bdb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4"
Oct 30 06:15:07.981543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.981512    5250 scope.go:122] "RemoveContainer" containerID="e7e9a3d9eb4333bd5b2f832d708cce50d5770d4829202631c95dd52a7836599e"
Oct 30 06:15:07.981833 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.981808    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6_odh-ai-gateway-infra(ff353edf-ad97-4352-a8b2-470fa965104c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" podUID="ff353edf-ad97-4352-a8b2-470fa965104c"
Oct 30 06:15:07.983111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.983094    5250 scope.go:122] "RemoveContainer" containerID="be5fcb5c1e1eea007e5dac986e235aef819b5ad96e2cbbfe848515b01dc82ab6"
Oct 30 06:15:07.983361 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.983327    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2_odh-ai-gateway-infra(259369fd-d911-4b38-96d2-ee3b43d3c075)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075"
Oct 30 06:15:07.984941 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.984921    5250 scope.go:122] "RemoveContainer" containerID="1cd30f877aa59eb96a8e8ab7c7b5fd7427bdebe6790615d6675861a65710eefe"
Oct 30 06:15:07.985219 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.985198    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9_odh-ai-gateway-infra(c9c63877-7c54-427d-b8bf-c0f1b22dcebb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb"
Oct 30 06:15:07.986838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.986787    5250 scope.go:122] "RemoveContainer" containerID="e80ab7cc63315f08e7a8b60f9ebc7ba035b12837a4d3f9a6d6b2c4d525d0d366"
Oct 30 06:15:07.987038 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.987015    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p_odh-ai-gateway-infra(bc73291b-82f1-4261-bc0b-8712d8a49558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558"
Oct 30 06:15:07.988533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.988508    5250 scope.go:122] "RemoveContainer" containerID="ea68c3d10b3ec16bd5d61b8d6bbfeb3272a7edc9e1d1f6fef59ba6e8e74ee9f4"
Oct 30 06:15:07.988764 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.988739    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9_odh-ai-gateway-infra(d09445ae-5f09-4c18-a37e-3af42038cd1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e"
Oct 30 06:15:07.990108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.990090    5250 scope.go:122] "RemoveContainer" containerID="e36f8bf952eb1fe0395b43013d717fd0f0cc942cf19680b1926901dd9bef0f46"
Oct 30 06:15:07.990353 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:07.990333    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-rrn6s_odh-ai-gateway-infra(ab62622d-84ad-4903-9be4-3b6ec51291f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3"
Oct 30 06:15:07.991447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.991426    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f" event={"ID":"8009548a-1020-4ebb-bb28-a363a1b6f9ee","Type":"ContainerStarted","Data":"f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a"}
Oct 30 06:15:07.991530 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.991503    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:07.991530 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:07.991519    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f" event={"ID":"8009548a-1020-4ebb-bb28-a363a1b6f9ee","Type":"ContainerStarted","Data":"22356e674f1cd400836617464aa629bca3f5b253dd5cdf4474636eee090b332c"}
Oct 30 06:15:08.055797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:08.055749    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f" podStartSLOduration=2.055734945 podStartE2EDuration="2.055734945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:08.050551858 +0000 UTC m=+2622.881589076" watchObservedRunningTime="2026-10-30 06:15:08.055734945 +0000 UTC m=+2622.886772162"
Oct 30 06:15:09.026424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:09.026389    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:09.570724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:09.570660    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:09.724166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:09.723472    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:09.730213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:09.726474    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"]
Oct 30 06:15:09.876301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:09.871075    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:10.867603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:10.867569    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:12.460547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:12.460513    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:14.177372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:14.177333    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:15.773926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:15.773893    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:16.274296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:16.274263    5250 scope.go:122] "RemoveContainer" containerID="e7e9a3d9eb4333bd5b2f832d708cce50d5770d4829202631c95dd52a7836599e"
Oct 30 06:15:17.123996 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:17.123961    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" event={"ID":"ff353edf-ad97-4352-a8b2-470fa965104c","Type":"ContainerStarted","Data":"ff5cebe538b0b7c1bf6ce5f0c231e180594160d8950529af171f8af58aea0574"}
Oct 30 06:15:17.274320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:17.274295    5250 scope.go:122] "RemoveContainer" containerID="e80ab7cc63315f08e7a8b60f9ebc7ba035b12837a4d3f9a6d6b2c4d525d0d366"
Oct 30 06:15:17.274481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:17.274461    5250 scope.go:122] "RemoveContainer" containerID="6ffd0f2d37f158fb53049b3fc5cd82a9ddfd147e809e445a2e6de04e9edff92d"
Oct 30 06:15:17.274636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:17.274617    5250 scope.go:122] "RemoveContainer" containerID="65e17cfcf029f065be7b15499e759b790be441738eb9ac7c9629378156bcfd5a"
Oct 30 06:15:17.475479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:17.475446    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:18.138167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.138131    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" event={"ID":"6fd00ce8-6874-4369-9022-1d1001fcbae3","Type":"ContainerStarted","Data":"3f928c82ecd12916d51fae53e4b5f535ace553ac99f3e64ecd3ab9d89731e918"}
Oct 30 06:15:18.140328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.140301    5250 generic.go:356] "Generic (PLEG): container finished" podID="ff353edf-ad97-4352-a8b2-470fa965104c" containerID="ff5cebe538b0b7c1bf6ce5f0c231e180594160d8950529af171f8af58aea0574" exitCode=60
Oct 30 06:15:18.140497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.140333    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" event={"ID":"ff353edf-ad97-4352-a8b2-470fa965104c","Type":"ContainerDied","Data":"ff5cebe538b0b7c1bf6ce5f0c231e180594160d8950529af171f8af58aea0574"}
Oct 30 06:15:18.140497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.140388    5250 scope.go:122] "RemoveContainer" containerID="e7e9a3d9eb4333bd5b2f832d708cce50d5770d4829202631c95dd52a7836599e"
Oct 30 06:15:18.140806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.140783    5250 scope.go:122] "RemoveContainer" containerID="ff5cebe538b0b7c1bf6ce5f0c231e180594160d8950529af171f8af58aea0574"
Oct 30 06:15:18.142640 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:18.141351    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6_odh-ai-gateway-infra(ff353edf-ad97-4352-a8b2-470fa965104c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" podUID="ff353edf-ad97-4352-a8b2-470fa965104c"
Oct 30 06:15:18.142832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.142777    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" event={"ID":"bc73291b-82f1-4261-bc0b-8712d8a49558","Type":"ContainerStarted","Data":"125a277445603311bca81fd9daced383270a29ebed9d4caa31769af6fe4406a4"}
Oct 30 06:15:18.145292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.145264    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" event={"ID":"f3be9b16-06d1-4ab5-a8e8-1825afb98f39","Type":"ContainerStarted","Data":"06f00dfea8975adfcd155e3a0c357f778789a323d179b0f12ebc3917329296b2"}
Oct 30 06:15:18.173639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.173612    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"]
Oct 30 06:15:18.273790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.273763    5250 scope.go:122] "RemoveContainer" containerID="be5fcb5c1e1eea007e5dac986e235aef819b5ad96e2cbbfe848515b01dc82ab6"
Oct 30 06:15:18.273936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.273917    5250 scope.go:122] "RemoveContainer" containerID="8de496c4884d46c24697e89dc4368a2f86026d4e7a1a1e8df0d849a48df1f89f"
Oct 30 06:15:18.274046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:18.274028    5250 scope.go:122] "RemoveContainer" containerID="00fc60f42e6268bca7d5fb01573878c193329d1d9db6b6bc70bc63d960ad98ed"
Oct 30 06:15:19.003384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.003357    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:15:19.160213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.160164    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" event={"ID":"8e3d2145-c914-4668-b07d-c7058b2d3f59","Type":"ContainerStarted","Data":"60ab550d54eee27ac14ef23690ab4a29e8c8948fd75696e7d5f329973d4f6de0"}
Oct 30 06:15:19.162793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.162768    5250 generic.go:356] "Generic (PLEG): container finished" podID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerID="3f928c82ecd12916d51fae53e4b5f535ace553ac99f3e64ecd3ab9d89731e918" exitCode=60
Oct 30 06:15:19.162930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.162813    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" event={"ID":"6fd00ce8-6874-4369-9022-1d1001fcbae3","Type":"ContainerDied","Data":"3f928c82ecd12916d51fae53e4b5f535ace553ac99f3e64ecd3ab9d89731e918"}
Oct 30 06:15:19.162930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.162883    5250 scope.go:122] "RemoveContainer" containerID="65e17cfcf029f065be7b15499e759b790be441738eb9ac7c9629378156bcfd5a"
Oct 30 06:15:19.163328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.163307    5250 scope.go:122] "RemoveContainer" containerID="3f928c82ecd12916d51fae53e4b5f535ace553ac99f3e64ecd3ab9d89731e918"
Oct 30 06:15:19.164086 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:19.163738    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b_odh-ai-gateway-infra(6fd00ce8-6874-4369-9022-1d1001fcbae3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3"
Oct 30 06:15:19.166548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.166527    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" event={"ID":"404e0b49-889c-4102-91fd-b989dcd5445c","Type":"ContainerStarted","Data":"90f49a0cc2422c1e41b40c6e8d933335d9143933cd5e9c577feff335ef503c4f"}
Oct 30 06:15:19.170857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.170835    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" event={"ID":"259369fd-d911-4b38-96d2-ee3b43d3c075","Type":"ContainerStarted","Data":"d6f8abc8539d04cd8260f13e500e2448e13bac904d63d8fc73b70b73d8fbc9f3"}
Oct 30 06:15:19.173196 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.173155    5250 generic.go:356] "Generic (PLEG): container finished" podID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerID="125a277445603311bca81fd9daced383270a29ebed9d4caa31769af6fe4406a4" exitCode=60
Oct 30 06:15:19.173297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.173204    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" event={"ID":"bc73291b-82f1-4261-bc0b-8712d8a49558","Type":"ContainerDied","Data":"125a277445603311bca81fd9daced383270a29ebed9d4caa31769af6fe4406a4"}
Oct 30 06:15:19.173556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.173539    5250 scope.go:122] "RemoveContainer" containerID="125a277445603311bca81fd9daced383270a29ebed9d4caa31769af6fe4406a4"
Oct 30 06:15:19.173860 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:19.173841    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p_odh-ai-gateway-infra(bc73291b-82f1-4261-bc0b-8712d8a49558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558"
Oct 30 06:15:19.175675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.175656    5250 generic.go:356] "Generic (PLEG): container finished" podID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerID="06f00dfea8975adfcd155e3a0c357f778789a323d179b0f12ebc3917329296b2" exitCode=60
Oct 30 06:15:19.175766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.175726    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" event={"ID":"f3be9b16-06d1-4ab5-a8e8-1825afb98f39","Type":"ContainerDied","Data":"06f00dfea8975adfcd155e3a0c357f778789a323d179b0f12ebc3917329296b2"}
Oct 30 06:15:19.176047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.176031    5250 scope.go:122] "RemoveContainer" containerID="06f00dfea8975adfcd155e3a0c357f778789a323d179b0f12ebc3917329296b2"
Oct 30 06:15:19.176367 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:19.176345    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb_odh-ai-gateway-infra(f3be9b16-06d1-4ab5-a8e8-1825afb98f39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39"
Oct 30 06:15:19.179914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.179898    5250 scope.go:122] "RemoveContainer" containerID="e80ab7cc63315f08e7a8b60f9ebc7ba035b12837a4d3f9a6d6b2c4d525d0d366"
Oct 30 06:15:19.195419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.195385    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"]
Oct 30 06:15:19.218258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.218228    5250 scope.go:122] "RemoveContainer" containerID="6ffd0f2d37f158fb53049b3fc5cd82a9ddfd147e809e445a2e6de04e9edff92d"
Oct 30 06:15:19.229743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.229717    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"]
Oct 30 06:15:19.238714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.238680    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:19.305665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.305635    5250 scope.go:122] "RemoveContainer" containerID="e36f8bf952eb1fe0395b43013d717fd0f0cc942cf19680b1926901dd9bef0f46"
Oct 30 06:15:19.307361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.307278    5250 scope.go:122] "RemoveContainer" containerID="db84471f566699317752eae6c6b340ce2acba9427a92287867f36742cd4e900d"
Oct 30 06:15:19.307867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.307850    5250 scope.go:122] "RemoveContainer" containerID="ea68c3d10b3ec16bd5d61b8d6bbfeb3272a7edc9e1d1f6fef59ba6e8e74ee9f4"
Oct 30 06:15:19.334365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.334340    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"]
Oct 30 06:15:19.423015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.422955    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:19.486984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.486958    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff353edf-ad97-4352-a8b2-470fa965104c-kube-api-access-6z9n5\" (UniqueName: \"kubernetes.io/projected/ff353edf-ad97-4352-a8b2-470fa965104c-kube-api-access-6z9n5\") pod \"ff353edf-ad97-4352-a8b2-470fa965104c\" (UID: \"ff353edf-ad97-4352-a8b2-470fa965104c\") "
Oct 30 06:15:19.487172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.487106    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff353edf-ad97-4352-a8b2-470fa965104c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff353edf-ad97-4352-a8b2-470fa965104c-maas-api-service-ca\") pod \"ff353edf-ad97-4352-a8b2-470fa965104c\" (UID: \"ff353edf-ad97-4352-a8b2-470fa965104c\") "
Oct 30 06:15:19.487480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.487455    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff353edf-ad97-4352-a8b2-470fa965104c-maas-api-service-ca" pod "ff353edf-ad97-4352-a8b2-470fa965104c" (UID: "ff353edf-ad97-4352-a8b2-470fa965104c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:19.487831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.487815    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff353edf-ad97-4352-a8b2-470fa965104c-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:19.569257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.560891    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff353edf-ad97-4352-a8b2-470fa965104c-kube-api-access-6z9n5" pod "ff353edf-ad97-4352-a8b2-470fa965104c" (UID: "ff353edf-ad97-4352-a8b2-470fa965104c"). InnerVolumeSpecName "kube-api-access-6z9n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:19.589843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:19.589802    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9n5\" (UniqueName: \"kubernetes.io/projected/ff353edf-ad97-4352-a8b2-470fa965104c-kube-api-access-6z9n5\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:20.187289 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.187256    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" event={"ID":"d01694ac-ebec-4c70-a10a-39f4de159d25","Type":"ContainerStarted","Data":"1dc86842e329f838d3bcbbf8513ca6646cac6e343b83a6e866291697ba02495d"}
Oct 30 06:15:20.189407 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.189384    5250 generic.go:356] "Generic (PLEG): container finished" podID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerID="60ab550d54eee27ac14ef23690ab4a29e8c8948fd75696e7d5f329973d4f6de0" exitCode=60
Oct 30 06:15:20.189566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.189456    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" event={"ID":"8e3d2145-c914-4668-b07d-c7058b2d3f59","Type":"ContainerDied","Data":"60ab550d54eee27ac14ef23690ab4a29e8c8948fd75696e7d5f329973d4f6de0"}
Oct 30 06:15:20.189566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.189505    5250 scope.go:122] "RemoveContainer" containerID="00fc60f42e6268bca7d5fb01573878c193329d1d9db6b6bc70bc63d960ad98ed"
Oct 30 06:15:20.189905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.189889    5250 scope.go:122] "RemoveContainer" containerID="60ab550d54eee27ac14ef23690ab4a29e8c8948fd75696e7d5f329973d4f6de0"
Oct 30 06:15:20.190330 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:20.190306    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h_odh-ai-gateway-infra(8e3d2145-c914-4668-b07d-c7058b2d3f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59"
Oct 30 06:15:20.193900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.193875    5250 generic.go:356] "Generic (PLEG): container finished" podID="404e0b49-889c-4102-91fd-b989dcd5445c" containerID="90f49a0cc2422c1e41b40c6e8d933335d9143933cd5e9c577feff335ef503c4f" exitCode=60
Oct 30 06:15:20.194029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.193942    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" event={"ID":"404e0b49-889c-4102-91fd-b989dcd5445c","Type":"ContainerDied","Data":"90f49a0cc2422c1e41b40c6e8d933335d9143933cd5e9c577feff335ef503c4f"}
Oct 30 06:15:20.194422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.194400    5250 scope.go:122] "RemoveContainer" containerID="90f49a0cc2422c1e41b40c6e8d933335d9143933cd5e9c577feff335ef503c4f"
Oct 30 06:15:20.194793 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:20.194770    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw_odh-ai-gateway-infra(404e0b49-889c-4102-91fd-b989dcd5445c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" podUID="404e0b49-889c-4102-91fd-b989dcd5445c"
Oct 30 06:15:20.195849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.195832    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"
Oct 30 06:15:20.195849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.195842    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6" event={"ID":"ff353edf-ad97-4352-a8b2-470fa965104c","Type":"ContainerDied","Data":"b5a801db38cc7790649a513234097123e985d92690982a713698372a1fe8d00b"}
Oct 30 06:15:20.198425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.198403    5250 generic.go:356] "Generic (PLEG): container finished" podID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerID="d6f8abc8539d04cd8260f13e500e2448e13bac904d63d8fc73b70b73d8fbc9f3" exitCode=60
Oct 30 06:15:20.198544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.198470    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" event={"ID":"259369fd-d911-4b38-96d2-ee3b43d3c075","Type":"ContainerDied","Data":"d6f8abc8539d04cd8260f13e500e2448e13bac904d63d8fc73b70b73d8fbc9f3"}
Oct 30 06:15:20.198826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.198808    5250 scope.go:122] "RemoveContainer" containerID="d6f8abc8539d04cd8260f13e500e2448e13bac904d63d8fc73b70b73d8fbc9f3"
Oct 30 06:15:20.199122 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:20.199099    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2_odh-ai-gateway-infra(259369fd-d911-4b38-96d2-ee3b43d3c075)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075"
Oct 30 06:15:20.204562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.204539    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" event={"ID":"d09445ae-5f09-4c18-a37e-3af42038cd1e","Type":"ContainerStarted","Data":"fc0c9c999aa086e969dcef0e6cc841ab10732dba266bc168dc6c1877b210bb4b"}
Oct 30 06:15:20.209085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.209064    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" event={"ID":"ab62622d-84ad-4903-9be4-3b6ec51291f3","Type":"ContainerStarted","Data":"705f4ea76d0b41e2f0a5afc285053b8a86f1636c41569b16210b51e665d425cf"}
Oct 30 06:15:20.209754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.209730    5250 scope.go:122] "RemoveContainer" containerID="8de496c4884d46c24697e89dc4368a2f86026d4e7a1a1e8df0d849a48df1f89f"
Oct 30 06:15:20.213362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.213136    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"]
Oct 30 06:15:20.253316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.253192    5250 scope.go:122] "RemoveContainer" containerID="ff5cebe538b0b7c1bf6ce5f0c231e180594160d8950529af171f8af58aea0574"
Oct 30 06:15:20.275423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.274641    5250 scope.go:122] "RemoveContainer" containerID="68c18e3345422b71c040dff56d7515374655b1851268eea9ea24d5f643eb535b"
Oct 30 06:15:20.279067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.277247    5250 scope.go:122] "RemoveContainer" containerID="1cd30f877aa59eb96a8e8ab7c7b5fd7427bdebe6790615d6675861a65710eefe"
Oct 30 06:15:20.279067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.278495    5250 scope.go:122] "RemoveContainer" containerID="ecb2d0486d2892b23c5453669b921ad37327976912541aa4a8810ca53283fa66"
Oct 30 06:15:20.298893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.291228    5250 scope.go:122] "RemoveContainer" containerID="be5fcb5c1e1eea007e5dac986e235aef819b5ad96e2cbbfe848515b01dc82ab6"
Oct 30 06:15:20.398423 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.398397    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"]
Oct 30 06:15:20.407139 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.407112    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2f1e4c-29889015-bwhf6"]
Oct 30 06:15:20.445411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.445380    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"]
Oct 30 06:15:20.479849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.479763    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:20.527218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.527153    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"]
Oct 30 06:15:20.626942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.626909    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd00ce8-6874-4369-9022-1d1001fcbae3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd00ce8-6874-4369-9022-1d1001fcbae3-maas-api-service-ca\") pod \"6fd00ce8-6874-4369-9022-1d1001fcbae3\" (UID: \"6fd00ce8-6874-4369-9022-1d1001fcbae3\") "
Oct 30 06:15:20.627091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.627021    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd00ce8-6874-4369-9022-1d1001fcbae3-kube-api-access-xd2ws\" (UniqueName: \"kubernetes.io/projected/6fd00ce8-6874-4369-9022-1d1001fcbae3-kube-api-access-xd2ws\") pod \"6fd00ce8-6874-4369-9022-1d1001fcbae3\" (UID: \"6fd00ce8-6874-4369-9022-1d1001fcbae3\") "
Oct 30 06:15:20.629129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.629102    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd00ce8-6874-4369-9022-1d1001fcbae3-maas-api-service-ca" pod "6fd00ce8-6874-4369-9022-1d1001fcbae3" (UID: "6fd00ce8-6874-4369-9022-1d1001fcbae3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:20.658885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.658805    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd00ce8-6874-4369-9022-1d1001fcbae3-kube-api-access-xd2ws" pod "6fd00ce8-6874-4369-9022-1d1001fcbae3" (UID: "6fd00ce8-6874-4369-9022-1d1001fcbae3"). InnerVolumeSpecName "kube-api-access-xd2ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:20.729438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.729360    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd00ce8-6874-4369-9022-1d1001fcbae3-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:20.729438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.729387    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2ws\" (UniqueName: \"kubernetes.io/projected/6fd00ce8-6874-4369-9022-1d1001fcbae3-kube-api-access-xd2ws\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:20.813648 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.813619    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:20.817534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.817512    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:20.932281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.932229    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-maas-api-service-ca\") pod \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\" (UID: \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\") "
Oct 30 06:15:20.932484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.932363    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-kube-api-access-fsb7n\" (UniqueName: \"kubernetes.io/projected/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-kube-api-access-fsb7n\") pod \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\" (UID: \"f3be9b16-06d1-4ab5-a8e8-1825afb98f39\") "
Oct 30 06:15:20.932484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.932446    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc73291b-82f1-4261-bc0b-8712d8a49558-kube-api-access-vdlfc\" (UniqueName: \"kubernetes.io/projected/bc73291b-82f1-4261-bc0b-8712d8a49558-kube-api-access-vdlfc\") pod \"bc73291b-82f1-4261-bc0b-8712d8a49558\" (UID: \"bc73291b-82f1-4261-bc0b-8712d8a49558\") "
Oct 30 06:15:20.932684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.932666    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc73291b-82f1-4261-bc0b-8712d8a49558-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc73291b-82f1-4261-bc0b-8712d8a49558-maas-api-service-ca\") pod \"bc73291b-82f1-4261-bc0b-8712d8a49558\" (UID: \"bc73291b-82f1-4261-bc0b-8712d8a49558\") "
Oct 30 06:15:20.932806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.932776    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-maas-api-service-ca" pod "f3be9b16-06d1-4ab5-a8e8-1825afb98f39" (UID: "f3be9b16-06d1-4ab5-a8e8-1825afb98f39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:20.933368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.933323    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:20.933511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.933421    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc73291b-82f1-4261-bc0b-8712d8a49558-maas-api-service-ca" pod "bc73291b-82f1-4261-bc0b-8712d8a49558" (UID: "bc73291b-82f1-4261-bc0b-8712d8a49558"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:20.936123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.936082    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc73291b-82f1-4261-bc0b-8712d8a49558-kube-api-access-vdlfc" pod "bc73291b-82f1-4261-bc0b-8712d8a49558" (UID: "bc73291b-82f1-4261-bc0b-8712d8a49558"). InnerVolumeSpecName "kube-api-access-vdlfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:20.936317 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:20.936284    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-kube-api-access-fsb7n" pod "f3be9b16-06d1-4ab5-a8e8-1825afb98f39" (UID: "f3be9b16-06d1-4ab5-a8e8-1825afb98f39"). InnerVolumeSpecName "kube-api-access-fsb7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:21.034884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.034804    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsb7n\" (UniqueName: \"kubernetes.io/projected/f3be9b16-06d1-4ab5-a8e8-1825afb98f39-kube-api-access-fsb7n\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.034884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.034835    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlfc\" (UniqueName: \"kubernetes.io/projected/bc73291b-82f1-4261-bc0b-8712d8a49558-kube-api-access-vdlfc\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.034884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.034845    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc73291b-82f1-4261-bc0b-8712d8a49558-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.130767 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.130714    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:21.229229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.229146    5250 generic.go:356] "Generic (PLEG): container finished" podID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerID="fc0c9c999aa086e969dcef0e6cc841ab10732dba266bc168dc6c1877b210bb4b" exitCode=60
Oct 30 06:15:21.229722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.229251    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" event={"ID":"d09445ae-5f09-4c18-a37e-3af42038cd1e","Type":"ContainerDied","Data":"fc0c9c999aa086e969dcef0e6cc841ab10732dba266bc168dc6c1877b210bb4b"}
Oct 30 06:15:21.229722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.229323    5250 scope.go:122] "RemoveContainer" containerID="ea68c3d10b3ec16bd5d61b8d6bbfeb3272a7edc9e1d1f6fef59ba6e8e74ee9f4"
Oct 30 06:15:21.229866 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.229772    5250 scope.go:122] "RemoveContainer" containerID="fc0c9c999aa086e969dcef0e6cc841ab10732dba266bc168dc6c1877b210bb4b"
Oct 30 06:15:21.230474 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:21.230251    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9_odh-ai-gateway-infra(d09445ae-5f09-4c18-a37e-3af42038cd1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e"
Oct 30 06:15:21.234809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.234784    5250 generic.go:356] "Generic (PLEG): container finished" podID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerID="705f4ea76d0b41e2f0a5afc285053b8a86f1636c41569b16210b51e665d425cf" exitCode=60
Oct 30 06:15:21.234938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.234866    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" event={"ID":"ab62622d-84ad-4903-9be4-3b6ec51291f3","Type":"ContainerDied","Data":"705f4ea76d0b41e2f0a5afc285053b8a86f1636c41569b16210b51e665d425cf"}
Oct 30 06:15:21.235471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.235446    5250 scope.go:122] "RemoveContainer" containerID="705f4ea76d0b41e2f0a5afc285053b8a86f1636c41569b16210b51e665d425cf"
Oct 30 06:15:21.236016 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:21.235863    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-rrn6s_odh-ai-gateway-infra(ab62622d-84ad-4903-9be4-3b6ec51291f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3"
Oct 30 06:15:21.236955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.236928    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb" event={"ID":"f3be9b16-06d1-4ab5-a8e8-1825afb98f39","Type":"ContainerDied","Data":"38b0fee18e1e90b17f3ec780ec7061e82ba18624d0f693b6679dba5a8ce931e4"}
Oct 30 06:15:21.237072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.237018    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"
Oct 30 06:15:21.243929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.243334    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"]
Oct 30 06:15:21.247048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.247022    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" event={"ID":"27ebfb55-80c6-4651-b46c-e588b914c30c","Type":"ContainerStarted","Data":"2dde54e2524157a3e43e060b15aae8301f47ec23988e9d19b4c851af1a6f8830"}
Oct 30 06:15:21.250413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.250391    5250 generic.go:356] "Generic (PLEG): container finished" podID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerID="1dc86842e329f838d3bcbbf8513ca6646cac6e343b83a6e866291697ba02495d" exitCode=60
Oct 30 06:15:21.250525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.250459    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" event={"ID":"d01694ac-ebec-4c70-a10a-39f4de159d25","Type":"ContainerDied","Data":"1dc86842e329f838d3bcbbf8513ca6646cac6e343b83a6e866291697ba02495d"}
Oct 30 06:15:21.250963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.250943    5250 scope.go:122] "RemoveContainer" containerID="e36f8bf952eb1fe0395b43013d717fd0f0cc942cf19680b1926901dd9bef0f46"
Oct 30 06:15:21.268392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.268362    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" event={"ID":"4c5783c3-b770-4ded-a301-7e607e09bdb4","Type":"ContainerStarted","Data":"b2b46b396bb929735fb8bb96c3cba29540a1c671802664bddd624e413973d10e"}
Oct 30 06:15:21.271862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.271788    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b" event={"ID":"6fd00ce8-6874-4369-9022-1d1001fcbae3","Type":"ContainerDied","Data":"da3e587970c92f79c5f00adadb7907c56c0528bbc9b4a7281d617dcf8a985571"}
Oct 30 06:15:21.272105 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.271797    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"
Oct 30 06:15:21.301997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.301974    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"
Oct 30 06:15:21.308863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.308841    5250 scope.go:122] "RemoveContainer" containerID="06f00dfea8975adfcd155e3a0c357f778789a323d179b0f12ebc3917329296b2"
Oct 30 06:15:21.326477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.325428    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" path="/var/lib/kubelet/pods/ff353edf-ad97-4352-a8b2-470fa965104c/volumes"
Oct 30 06:15:21.327363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.326814    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" event={"ID":"c9c63877-7c54-427d-b8bf-c0f1b22dcebb","Type":"ContainerStarted","Data":"e42a8259fe85c7d086db4c009cd32cb4bed25c08fbdfca631cb212c33ffb9259"}
Oct 30 06:15:21.327363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.326891    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p" event={"ID":"bc73291b-82f1-4261-bc0b-8712d8a49558","Type":"ContainerDied","Data":"6abac56512a3a36e4bec2c88031d760a1fe22b4ac9dbd54d4d3adf5b55a0a87a"}
Oct 30 06:15:21.336202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.336163    5250 scope.go:122] "RemoveContainer" containerID="db84471f566699317752eae6c6b340ce2acba9427a92287867f36742cd4e900d"
Oct 30 06:15:21.372308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.371380    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"]
Oct 30 06:15:21.376647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.376611    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6b8f87-29889015-2swlb"]
Oct 30 06:15:21.387030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.385285    5250 scope.go:122] "RemoveContainer" containerID="3f928c82ecd12916d51fae53e4b5f535ace553ac99f3e64ecd3ab9d89731e918"
Oct 30 06:15:21.405443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.405420    5250 scope.go:122] "RemoveContainer" containerID="125a277445603311bca81fd9daced383270a29ebed9d4caa31769af6fe4406a4"
Oct 30 06:15:21.446038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.446009    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"]
Oct 30 06:15:21.537354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.537277    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:21.557277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.557247    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"]
Oct 30 06:15:21.567234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.567164    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"]
Oct 30 06:15:21.581765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.580886    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95498c-29889015-cpq6p"]
Oct 30 06:15:21.635862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.635835    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:21.643499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.643478    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:21.647630 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.647610    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:21.654932 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.654907    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01694ac-ebec-4c70-a10a-39f4de159d25-kube-api-access-s2dnm\" (UniqueName: \"kubernetes.io/projected/d01694ac-ebec-4c70-a10a-39f4de159d25-kube-api-access-s2dnm\") pod \"d01694ac-ebec-4c70-a10a-39f4de159d25\" (UID: \"d01694ac-ebec-4c70-a10a-39f4de159d25\") "
Oct 30 06:15:21.655098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.655086    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d01694ac-ebec-4c70-a10a-39f4de159d25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01694ac-ebec-4c70-a10a-39f4de159d25-maas-api-service-ca\") pod \"d01694ac-ebec-4c70-a10a-39f4de159d25\" (UID: \"d01694ac-ebec-4c70-a10a-39f4de159d25\") "
Oct 30 06:15:21.656289 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.656248    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01694ac-ebec-4c70-a10a-39f4de159d25-maas-api-service-ca" pod "d01694ac-ebec-4c70-a10a-39f4de159d25" (UID: "d01694ac-ebec-4c70-a10a-39f4de159d25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:21.658437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.658410    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01694ac-ebec-4c70-a10a-39f4de159d25-kube-api-access-s2dnm" pod "d01694ac-ebec-4c70-a10a-39f4de159d25" (UID: "d01694ac-ebec-4c70-a10a-39f4de159d25"). InnerVolumeSpecName "kube-api-access-s2dnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:21.726564 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.726535    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"]
Oct 30 06:15:21.731508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.731428    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889015-glb2b"]
Oct 30 06:15:21.756849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.756822    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259369fd-d911-4b38-96d2-ee3b43d3c075-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259369fd-d911-4b38-96d2-ee3b43d3c075-maas-api-service-ca\") pod \"259369fd-d911-4b38-96d2-ee3b43d3c075\" (UID: \"259369fd-d911-4b38-96d2-ee3b43d3c075\") "
Oct 30 06:15:21.756990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.756940    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259369fd-d911-4b38-96d2-ee3b43d3c075-kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/259369fd-d911-4b38-96d2-ee3b43d3c075-kube-api-access-rxxjf\") pod \"259369fd-d911-4b38-96d2-ee3b43d3c075\" (UID: \"259369fd-d911-4b38-96d2-ee3b43d3c075\") "
Oct 30 06:15:21.757114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.757099    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3d2145-c914-4668-b07d-c7058b2d3f59-kube-api-access-tv6gz\" (UniqueName: \"kubernetes.io/projected/8e3d2145-c914-4668-b07d-c7058b2d3f59-kube-api-access-tv6gz\") pod \"8e3d2145-c914-4668-b07d-c7058b2d3f59\" (UID: \"8e3d2145-c914-4668-b07d-c7058b2d3f59\") "
Oct 30 06:15:21.757191 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.757147    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3d2145-c914-4668-b07d-c7058b2d3f59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d2145-c914-4668-b07d-c7058b2d3f59-maas-api-service-ca\") pod \"8e3d2145-c914-4668-b07d-c7058b2d3f59\" (UID: \"8e3d2145-c914-4668-b07d-c7058b2d3f59\") "
Oct 30 06:15:21.757276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.757222    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259369fd-d911-4b38-96d2-ee3b43d3c075-maas-api-service-ca" pod "259369fd-d911-4b38-96d2-ee3b43d3c075" (UID: "259369fd-d911-4b38-96d2-ee3b43d3c075"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:21.757338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.757294    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404e0b49-889c-4102-91fd-b989dcd5445c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/404e0b49-889c-4102-91fd-b989dcd5445c-maas-api-service-ca\") pod \"404e0b49-889c-4102-91fd-b989dcd5445c\" (UID: \"404e0b49-889c-4102-91fd-b989dcd5445c\") "
Oct 30 06:15:21.757396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.757372    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404e0b49-889c-4102-91fd-b989dcd5445c-kube-api-access-mxwv7\" (UniqueName: \"kubernetes.io/projected/404e0b49-889c-4102-91fd-b989dcd5445c-kube-api-access-mxwv7\") pod \"404e0b49-889c-4102-91fd-b989dcd5445c\" (UID: \"404e0b49-889c-4102-91fd-b989dcd5445c\") "
Oct 30 06:15:21.757592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.757566    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3d2145-c914-4668-b07d-c7058b2d3f59-maas-api-service-ca" pod "8e3d2145-c914-4668-b07d-c7058b2d3f59" (UID: "8e3d2145-c914-4668-b07d-c7058b2d3f59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:21.757826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.757803    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404e0b49-889c-4102-91fd-b989dcd5445c-maas-api-service-ca" pod "404e0b49-889c-4102-91fd-b989dcd5445c" (UID: "404e0b49-889c-4102-91fd-b989dcd5445c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:21.758211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.758194    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259369fd-d911-4b38-96d2-ee3b43d3c075-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.758523 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.758498    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01694ac-ebec-4c70-a10a-39f4de159d25-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.758627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.758531    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d2145-c914-4668-b07d-c7058b2d3f59-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.758627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.758555    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/404e0b49-889c-4102-91fd-b989dcd5445c-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.758627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.758575    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2dnm\" (UniqueName: \"kubernetes.io/projected/d01694ac-ebec-4c70-a10a-39f4de159d25-kube-api-access-s2dnm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.760979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.760074    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404e0b49-889c-4102-91fd-b989dcd5445c-kube-api-access-mxwv7" pod "404e0b49-889c-4102-91fd-b989dcd5445c" (UID: "404e0b49-889c-4102-91fd-b989dcd5445c"). InnerVolumeSpecName "kube-api-access-mxwv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:21.760979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.760206    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259369fd-d911-4b38-96d2-ee3b43d3c075-kube-api-access-rxxjf" pod "259369fd-d911-4b38-96d2-ee3b43d3c075" (UID: "259369fd-d911-4b38-96d2-ee3b43d3c075"). InnerVolumeSpecName "kube-api-access-rxxjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:21.760979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.760676    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3d2145-c914-4668-b07d-c7058b2d3f59-kube-api-access-tv6gz" pod "8e3d2145-c914-4668-b07d-c7058b2d3f59" (UID: "8e3d2145-c914-4668-b07d-c7058b2d3f59"). InnerVolumeSpecName "kube-api-access-tv6gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:21.860510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.860433    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/259369fd-d911-4b38-96d2-ee3b43d3c075-kube-api-access-rxxjf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.860799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.860779    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6gz\" (UniqueName: \"kubernetes.io/projected/8e3d2145-c914-4668-b07d-c7058b2d3f59-kube-api-access-tv6gz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:21.860930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:21.860909    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxwv7\" (UniqueName: \"kubernetes.io/projected/404e0b49-889c-4102-91fd-b989dcd5445c-kube-api-access-mxwv7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:22.312836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.312805    5250 generic.go:356] "Generic (PLEG): container finished" podID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerID="b2b46b396bb929735fb8bb96c3cba29540a1c671802664bddd624e413973d10e" exitCode=60
Oct 30 06:15:22.313369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.312883    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" event={"ID":"4c5783c3-b770-4ded-a301-7e607e09bdb4","Type":"ContainerDied","Data":"b2b46b396bb929735fb8bb96c3cba29540a1c671802664bddd624e413973d10e"}
Oct 30 06:15:22.313369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.312961    5250 scope.go:122] "RemoveContainer" containerID="68c18e3345422b71c040dff56d7515374655b1851268eea9ea24d5f643eb535b"
Oct 30 06:15:22.313369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.313332    5250 scope.go:122] "RemoveContainer" containerID="b2b46b396bb929735fb8bb96c3cba29540a1c671802664bddd624e413973d10e"
Oct 30 06:15:22.313706 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:22.313681    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq_odh-ai-gateway-infra(4c5783c3-b770-4ded-a301-7e607e09bdb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4"
Oct 30 06:15:22.315576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.315551    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw" event={"ID":"404e0b49-889c-4102-91fd-b989dcd5445c","Type":"ContainerDied","Data":"3f9a81516c5fc76bf4c7d49e0b01f97dde26170726149f2141be78a83f0e54bf"}
Oct 30 06:15:22.315685 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.315559    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"
Oct 30 06:15:22.316983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.316958    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"
Oct 30 06:15:22.317102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.316982    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2" event={"ID":"259369fd-d911-4b38-96d2-ee3b43d3c075","Type":"ContainerDied","Data":"0fe63dec3b003dcc3eb3b64e7518606745ac80bc7b28be70c489d583a1044e1d"}
Oct 30 06:15:22.319364 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.319341    5250 generic.go:356] "Generic (PLEG): container finished" podID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerID="e42a8259fe85c7d086db4c009cd32cb4bed25c08fbdfca631cb212c33ffb9259" exitCode=60
Oct 30 06:15:22.319464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.319404    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" event={"ID":"c9c63877-7c54-427d-b8bf-c0f1b22dcebb","Type":"ContainerDied","Data":"e42a8259fe85c7d086db4c009cd32cb4bed25c08fbdfca631cb212c33ffb9259"}
Oct 30 06:15:22.319777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.319760    5250 scope.go:122] "RemoveContainer" containerID="e42a8259fe85c7d086db4c009cd32cb4bed25c08fbdfca631cb212c33ffb9259"
Oct 30 06:15:22.320064 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:22.320037    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9_odh-ai-gateway-infra(c9c63877-7c54-427d-b8bf-c0f1b22dcebb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb"
Oct 30 06:15:22.329661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.329633    5250 generic.go:356] "Generic (PLEG): container finished" podID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerID="2dde54e2524157a3e43e060b15aae8301f47ec23988e9d19b4c851af1a6f8830" exitCode=60
Oct 30 06:15:22.329744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.329718    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" event={"ID":"27ebfb55-80c6-4651-b46c-e588b914c30c","Type":"ContainerDied","Data":"2dde54e2524157a3e43e060b15aae8301f47ec23988e9d19b4c851af1a6f8830"}
Oct 30 06:15:22.330130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.330110    5250 scope.go:122] "RemoveContainer" containerID="2dde54e2524157a3e43e060b15aae8301f47ec23988e9d19b4c851af1a6f8830"
Oct 30 06:15:22.330538 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:22.330510    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px_odh-ai-gateway-infra(27ebfb55-80c6-4651-b46c-e588b914c30c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c"
Oct 30 06:15:22.331229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.331204    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"
Oct 30 06:15:22.331319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.331235    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9" event={"ID":"d01694ac-ebec-4c70-a10a-39f4de159d25","Type":"ContainerDied","Data":"e5a5379f1c80b70bf4999f07e37709fbd8fba6682cf1cf22b25eefe183b9a987"}
Oct 30 06:15:22.331474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.331390    5250 scope.go:122] "RemoveContainer" containerID="90f49a0cc2422c1e41b40c6e8d933335d9143933cd5e9c577feff335ef503c4f"
Oct 30 06:15:22.333709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.333683    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h" event={"ID":"8e3d2145-c914-4668-b07d-c7058b2d3f59","Type":"ContainerDied","Data":"db56b0fefe04c9dc4da62da02ce7a91986d17ab338926ab5b2975bda133dfe32"}
Oct 30 06:15:22.333813 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.333744    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"
Oct 30 06:15:22.358307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.358280    5250 scope.go:122] "RemoveContainer" containerID="d6f8abc8539d04cd8260f13e500e2448e13bac904d63d8fc73b70b73d8fbc9f3"
Oct 30 06:15:22.372640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.372616    5250 scope.go:122] "RemoveContainer" containerID="1cd30f877aa59eb96a8e8ab7c7b5fd7427bdebe6790615d6675861a65710eefe"
Oct 30 06:15:22.390471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.390447    5250 scope.go:122] "RemoveContainer" containerID="ecb2d0486d2892b23c5453669b921ad37327976912541aa4a8810ca53283fa66"
Oct 30 06:15:22.408434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.408265    5250 scope.go:122] "RemoveContainer" containerID="1dc86842e329f838d3bcbbf8513ca6646cac6e343b83a6e866291697ba02495d"
Oct 30 06:15:22.450345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.450303    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"]
Oct 30 06:15:22.464454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.463748    5250 scope.go:122] "RemoveContainer" containerID="60ab550d54eee27ac14ef23690ab4a29e8c8948fd75696e7d5f329973d4f6de0"
Oct 30 06:15:22.502279 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.502226    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"]
Oct 30 06:15:22.509574 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.509412    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889015-25tn9"]
Oct 30 06:15:22.539911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.539877    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"]
Oct 30 06:15:22.546002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.545973    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"]
Oct 30 06:15:22.551239 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.551169    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:22.552896 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.552869    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29889015-t28g2"]
Oct 30 06:15:22.605758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.605733    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:22.615706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.615679    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"]
Oct 30 06:15:22.625751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.625727    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1b4b7a-29889015-trpzw"]
Oct 30 06:15:22.670865 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.670833    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"]
Oct 30 06:15:22.673193 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.673150    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab62622d-84ad-4903-9be4-3b6ec51291f3-kube-api-access-fbv8l\" (UniqueName: \"kubernetes.io/projected/ab62622d-84ad-4903-9be4-3b6ec51291f3-kube-api-access-fbv8l\") pod \"ab62622d-84ad-4903-9be4-3b6ec51291f3\" (UID: \"ab62622d-84ad-4903-9be4-3b6ec51291f3\") "
Oct 30 06:15:22.673394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.673377    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab62622d-84ad-4903-9be4-3b6ec51291f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62622d-84ad-4903-9be4-3b6ec51291f3-maas-api-service-ca\") pod \"ab62622d-84ad-4903-9be4-3b6ec51291f3\" (UID: \"ab62622d-84ad-4903-9be4-3b6ec51291f3\") "
Oct 30 06:15:22.673680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.673658    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab62622d-84ad-4903-9be4-3b6ec51291f3-maas-api-service-ca" pod "ab62622d-84ad-4903-9be4-3b6ec51291f3" (UID: "ab62622d-84ad-4903-9be4-3b6ec51291f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:22.673995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.673976    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62622d-84ad-4903-9be4-3b6ec51291f3-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:22.675532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.675511    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab62622d-84ad-4903-9be4-3b6ec51291f3-kube-api-access-fbv8l" pod "ab62622d-84ad-4903-9be4-3b6ec51291f3" (UID: "ab62622d-84ad-4903-9be4-3b6ec51291f3"). InnerVolumeSpecName "kube-api-access-fbv8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:22.736378 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.736335    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"]
Oct 30 06:15:22.741058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.741027    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-080391-29889015-ncz6h"]
Oct 30 06:15:22.774989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.774943    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09445ae-5f09-4c18-a37e-3af42038cd1e-kube-api-access-k88gj\" (UniqueName: \"kubernetes.io/projected/d09445ae-5f09-4c18-a37e-3af42038cd1e-kube-api-access-k88gj\") pod \"d09445ae-5f09-4c18-a37e-3af42038cd1e\" (UID: \"d09445ae-5f09-4c18-a37e-3af42038cd1e\") "
Oct 30 06:15:22.775157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.775100    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09445ae-5f09-4c18-a37e-3af42038cd1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09445ae-5f09-4c18-a37e-3af42038cd1e-maas-api-service-ca\") pod \"d09445ae-5f09-4c18-a37e-3af42038cd1e\" (UID: \"d09445ae-5f09-4c18-a37e-3af42038cd1e\") "
Oct 30 06:15:22.775918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.775891    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbv8l\" (UniqueName: \"kubernetes.io/projected/ab62622d-84ad-4903-9be4-3b6ec51291f3-kube-api-access-fbv8l\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:22.776143 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.776114    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09445ae-5f09-4c18-a37e-3af42038cd1e-maas-api-service-ca" pod "d09445ae-5f09-4c18-a37e-3af42038cd1e" (UID: "d09445ae-5f09-4c18-a37e-3af42038cd1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:22.777694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.777668    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09445ae-5f09-4c18-a37e-3af42038cd1e-kube-api-access-k88gj" pod "d09445ae-5f09-4c18-a37e-3af42038cd1e" (UID: "d09445ae-5f09-4c18-a37e-3af42038cd1e"). InnerVolumeSpecName "kube-api-access-k88gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:22.877086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.876998    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09445ae-5f09-4c18-a37e-3af42038cd1e-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:22.877086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:22.877028    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88gj\" (UniqueName: \"kubernetes.io/projected/d09445ae-5f09-4c18-a37e-3af42038cd1e-kube-api-access-k88gj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:23.287970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.287932    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" path="/var/lib/kubelet/pods/259369fd-d911-4b38-96d2-ee3b43d3c075/volumes"
Oct 30 06:15:23.288652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.288626    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" path="/var/lib/kubelet/pods/404e0b49-889c-4102-91fd-b989dcd5445c/volumes"
Oct 30 06:15:23.289288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.289267    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" path="/var/lib/kubelet/pods/6fd00ce8-6874-4369-9022-1d1001fcbae3/volumes"
Oct 30 06:15:23.290334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.290313    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" path="/var/lib/kubelet/pods/8e3d2145-c914-4668-b07d-c7058b2d3f59/volumes"
Oct 30 06:15:23.290918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.290895    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" path="/var/lib/kubelet/pods/bc73291b-82f1-4261-bc0b-8712d8a49558/volumes"
Oct 30 06:15:23.291525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.291505    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" path="/var/lib/kubelet/pods/d01694ac-ebec-4c70-a10a-39f4de159d25/volumes"
Oct 30 06:15:23.292579 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.292523    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" path="/var/lib/kubelet/pods/f3be9b16-06d1-4ab5-a8e8-1825afb98f39/volumes"
Oct 30 06:15:23.375351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.375327    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"
Oct 30 06:15:23.375950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.375349    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9" event={"ID":"d09445ae-5f09-4c18-a37e-3af42038cd1e","Type":"ContainerDied","Data":"634ef52a7cb0792e70522ce848b1ee6dbcc7e02bc0b1fe8273fc73b02c28c1b3"}
Oct 30 06:15:23.375950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.375465    5250 scope.go:122] "RemoveContainer" containerID="fc0c9c999aa086e969dcef0e6cc841ab10732dba266bc168dc6c1877b210bb4b"
Oct 30 06:15:23.379820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.379794    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:23.380463 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.380442    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s" event={"ID":"ab62622d-84ad-4903-9be4-3b6ec51291f3","Type":"ContainerDied","Data":"e5c0165f66200ffed88f262333df5d4b7c5d7e0c128bfe3f1e2e13ec609894a2"}
Oct 30 06:15:23.380590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.380481    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"
Oct 30 06:15:23.426775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.426747    5250 scope.go:122] "RemoveContainer" containerID="705f4ea76d0b41e2f0a5afc285053b8a86f1636c41569b16210b51e665d425cf"
Oct 30 06:15:23.568801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.568747    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"]
Oct 30 06:15:23.574365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.574332    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-rrn6s"]
Oct 30 06:15:23.602066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.601992    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"]
Oct 30 06:15:23.610100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.610072    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29889015-gqqx9"]
Oct 30 06:15:23.614040 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.614020    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:23.683673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.683638    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:23.700412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.700385    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:23.802506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.802472    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5783c3-b770-4ded-a301-7e607e09bdb4-kube-api-access-8p9ff\" (UniqueName: \"kubernetes.io/projected/4c5783c3-b770-4ded-a301-7e607e09bdb4-kube-api-access-8p9ff\") pod \"4c5783c3-b770-4ded-a301-7e607e09bdb4\" (UID: \"4c5783c3-b770-4ded-a301-7e607e09bdb4\") "
Oct 30 06:15:23.802703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.802586    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5783c3-b770-4ded-a301-7e607e09bdb4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5783c3-b770-4ded-a301-7e607e09bdb4-maas-api-service-ca\") pod \"4c5783c3-b770-4ded-a301-7e607e09bdb4\" (UID: \"4c5783c3-b770-4ded-a301-7e607e09bdb4\") "
Oct 30 06:15:23.802703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.802660    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-maas-api-service-ca\") pod \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\" (UID: \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\") "
Oct 30 06:15:23.802826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.802799    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-kube-api-access-s75km\" (UniqueName: \"kubernetes.io/projected/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-kube-api-access-s75km\") pod \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\" (UID: \"c9c63877-7c54-427d-b8bf-c0f1b22dcebb\") "
Oct 30 06:15:23.803050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.803019    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5783c3-b770-4ded-a301-7e607e09bdb4-maas-api-service-ca" pod "4c5783c3-b770-4ded-a301-7e607e09bdb4" (UID: "4c5783c3-b770-4ded-a301-7e607e09bdb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:23.803159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.803074    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-maas-api-service-ca" pod "c9c63877-7c54-427d-b8bf-c0f1b22dcebb" (UID: "c9c63877-7c54-427d-b8bf-c0f1b22dcebb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:23.803159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.802829    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ebfb55-80c6-4651-b46c-e588b914c30c-kube-api-access-jwfph\" (UniqueName: \"kubernetes.io/projected/27ebfb55-80c6-4651-b46c-e588b914c30c-kube-api-access-jwfph\") pod \"27ebfb55-80c6-4651-b46c-e588b914c30c\" (UID: \"27ebfb55-80c6-4651-b46c-e588b914c30c\") "
Oct 30 06:15:23.804511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.804471    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ebfb55-80c6-4651-b46c-e588b914c30c-maas-api-service-ca" pod "27ebfb55-80c6-4651-b46c-e588b914c30c" (UID: "27ebfb55-80c6-4651-b46c-e588b914c30c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:23.804711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.804659    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ebfb55-80c6-4651-b46c-e588b914c30c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfb55-80c6-4651-b46c-e588b914c30c-maas-api-service-ca\") pod \"27ebfb55-80c6-4651-b46c-e588b914c30c\" (UID: \"27ebfb55-80c6-4651-b46c-e588b914c30c\") "
Oct 30 06:15:23.805824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.805800    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5783c3-b770-4ded-a301-7e607e09bdb4-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:23.805988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.805967    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:23.806093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.805990    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfb55-80c6-4651-b46c-e588b914c30c-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:23.806093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.805812    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-kube-api-access-s75km" pod "c9c63877-7c54-427d-b8bf-c0f1b22dcebb" (UID: "c9c63877-7c54-427d-b8bf-c0f1b22dcebb"). InnerVolumeSpecName "kube-api-access-s75km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:23.806093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.806007    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ebfb55-80c6-4651-b46c-e588b914c30c-kube-api-access-jwfph" pod "27ebfb55-80c6-4651-b46c-e588b914c30c" (UID: "27ebfb55-80c6-4651-b46c-e588b914c30c"). InnerVolumeSpecName "kube-api-access-jwfph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:23.806288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.806145    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5783c3-b770-4ded-a301-7e607e09bdb4-kube-api-access-8p9ff" pod "4c5783c3-b770-4ded-a301-7e607e09bdb4" (UID: "4c5783c3-b770-4ded-a301-7e607e09bdb4"). InnerVolumeSpecName "kube-api-access-8p9ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:23.907963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.907924    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9ff\" (UniqueName: \"kubernetes.io/projected/4c5783c3-b770-4ded-a301-7e607e09bdb4-kube-api-access-8p9ff\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:23.907963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.907959    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s75km\" (UniqueName: \"kubernetes.io/projected/c9c63877-7c54-427d-b8bf-c0f1b22dcebb-kube-api-access-s75km\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:23.908168 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:23.907974    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfph\" (UniqueName: \"kubernetes.io/projected/27ebfb55-80c6-4651-b46c-e588b914c30c-kube-api-access-jwfph\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:24.395968 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.395934    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9" event={"ID":"c9c63877-7c54-427d-b8bf-c0f1b22dcebb","Type":"ContainerDied","Data":"bf2b5741880a268b03e2eaa74776516faf20fbb2a90ac23e30b198ea23e6acdf"}
Oct 30 06:15:24.396580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.395972    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"
Oct 30 06:15:24.396580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.396010    5250 scope.go:122] "RemoveContainer" containerID="e42a8259fe85c7d086db4c009cd32cb4bed25c08fbdfca631cb212c33ffb9259"
Oct 30 06:15:24.400612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.400587    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"
Oct 30 06:15:24.400954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.400603    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px" event={"ID":"27ebfb55-80c6-4651-b46c-e588b914c30c","Type":"ContainerDied","Data":"1d4bc9e451eda2931f26cf46b3641b84776681ec22ab35197ccf2df7157f1ab0"}
Oct 30 06:15:24.402356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.402332    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"
Oct 30 06:15:24.402464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.402427    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq" event={"ID":"4c5783c3-b770-4ded-a301-7e607e09bdb4","Type":"ContainerDied","Data":"a3986e4db0bfae6031e9405a13362f1f820ba1df01c87144c1e680fda3407105"}
Oct 30 06:15:24.431956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.431787    5250 scope.go:122] "RemoveContainer" containerID="2dde54e2524157a3e43e060b15aae8301f47ec23988e9d19b4c851af1a6f8830"
Oct 30 06:15:24.444538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.444518    5250 scope.go:122] "RemoveContainer" containerID="b2b46b396bb929735fb8bb96c3cba29540a1c671802664bddd624e413973d10e"
Oct 30 06:15:24.471254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.471230    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"]
Oct 30 06:15:24.475612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.475587    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c2be21-29889015-pn9r9"]
Oct 30 06:15:24.511618 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.511593    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"]
Oct 30 06:15:24.518246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.518219    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-370dfa-29889015-8r9px"]
Oct 30 06:15:24.548007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.547979    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"]
Oct 30 06:15:24.552578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.552555    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889015-l75vq"]
Oct 30 06:15:24.761482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:24.761390    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65" podUID="b3c436f8-7a93-4543-952e-35149302152f" containerName="main" containerID="cri-o://dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7" gracePeriod=45
Oct 30 06:15:25.058883 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.058859    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:15:25.124792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.124759    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-tmp-dir\") pod \"b3c436f8-7a93-4543-952e-35149302152f\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") "
Oct 30 06:15:25.125012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.124997    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c436f8-7a93-4543-952e-35149302152f-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c436f8-7a93-4543-952e-35149302152f-tls-certs\") pod \"b3c436f8-7a93-4543-952e-35149302152f\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") "
Oct 30 06:15:25.125118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.125086    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-tmp-dir" pod "b3c436f8-7a93-4543-952e-35149302152f" (UID: "b3c436f8-7a93-4543-952e-35149302152f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:25.125397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.125374    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-dshm\") pod \"b3c436f8-7a93-4543-952e-35149302152f\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") "
Oct 30 06:15:25.125627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.125608    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c436f8-7a93-4543-952e-35149302152f-kube-api-access-ffsfc\" (UniqueName: \"kubernetes.io/projected/b3c436f8-7a93-4543-952e-35149302152f-kube-api-access-ffsfc\") pod \"b3c436f8-7a93-4543-952e-35149302152f\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") "
Oct 30 06:15:25.125765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.125750    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-model-cache\") pod \"b3c436f8-7a93-4543-952e-35149302152f\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") "
Oct 30 06:15:25.125959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.125944    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-home\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-home\") pod \"b3c436f8-7a93-4543-952e-35149302152f\" (UID: \"b3c436f8-7a93-4543-952e-35149302152f\") "
Oct 30 06:15:25.126129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.126105    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-home" pod "b3c436f8-7a93-4543-952e-35149302152f" (UID: "b3c436f8-7a93-4543-952e-35149302152f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:25.126281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.126261    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-model-cache" pod "b3c436f8-7a93-4543-952e-35149302152f" (UID: "b3c436f8-7a93-4543-952e-35149302152f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:25.127940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.126709    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:25.127940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.126737    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:25.127940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.126751    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:25.129355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.129016    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-dshm" pod "b3c436f8-7a93-4543-952e-35149302152f" (UID: "b3c436f8-7a93-4543-952e-35149302152f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:25.129591 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.129460    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c436f8-7a93-4543-952e-35149302152f-kube-api-access-ffsfc" pod "b3c436f8-7a93-4543-952e-35149302152f" (UID: "b3c436f8-7a93-4543-952e-35149302152f"). InnerVolumeSpecName "kube-api-access-ffsfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:25.132060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.132032    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c436f8-7a93-4543-952e-35149302152f-tls-certs" pod "b3c436f8-7a93-4543-952e-35149302152f" (UID: "b3c436f8-7a93-4543-952e-35149302152f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:15:25.156092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.156067    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:25.228559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.228518    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c436f8-7a93-4543-952e-35149302152f-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:25.228559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.228552    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffsfc\" (UniqueName: \"kubernetes.io/projected/b3c436f8-7a93-4543-952e-35149302152f-kube-api-access-ffsfc\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:25.228559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.228564    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c436f8-7a93-4543-952e-35149302152f-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:25.284983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.284913    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" path="/var/lib/kubelet/pods/27ebfb55-80c6-4651-b46c-e588b914c30c/volumes"
Oct 30 06:15:25.285408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.285392    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" path="/var/lib/kubelet/pods/4c5783c3-b770-4ded-a301-7e607e09bdb4/volumes"
Oct 30 06:15:25.285792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.285777    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" path="/var/lib/kubelet/pods/ab62622d-84ad-4903-9be4-3b6ec51291f3/volumes"
Oct 30 06:15:25.286578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.286558    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" path="/var/lib/kubelet/pods/c9c63877-7c54-427d-b8bf-c0f1b22dcebb/volumes"
Oct 30 06:15:25.286975 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.286961    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" path="/var/lib/kubelet/pods/d09445ae-5f09-4c18-a37e-3af42038cd1e/volumes"
Oct 30 06:15:25.414809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.414776    5250 generic.go:356] "Generic (PLEG): container finished" podID="b3c436f8-7a93-4543-952e-35149302152f" containerID="dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7" exitCode=0
Oct 30 06:15:25.415365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.414832    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"
Oct 30 06:15:25.415365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.414857    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65" event={"ID":"b3c436f8-7a93-4543-952e-35149302152f","Type":"ContainerDied","Data":"dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7"}
Oct 30 06:15:25.415365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.414901    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65" event={"ID":"b3c436f8-7a93-4543-952e-35149302152f","Type":"ContainerDied","Data":"eef31063578ad2fb5ba0bcb53f8460aad63d4a4c6bd62a105520598b2e1101ae"}
Oct 30 06:15:25.415365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.414921    5250 scope.go:122] "RemoveContainer" containerID="dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7"
Oct 30 06:15:25.426485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.426469    5250 scope.go:122] "RemoveContainer" containerID="dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7"
Oct 30 06:15:25.426724 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:25.426704    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7\": container with ID starting with dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7 not found: ID does not exist" containerID="dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7"
Oct 30 06:15:25.426789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.426732    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7"} err="failed to get container status \"dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7\": rpc error: code = NotFound desc = could not find container \"dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7\": container with ID starting with dc43e17721a33a72072b6596f3260bbc5b97047239ebbf9ed3c16ebb26e6f9c7 not found: ID does not exist"
Oct 30 06:15:25.464793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.464722    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"]
Oct 30 06:15:25.469689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:25.469653    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/rate-test-model-3abe42-kserve-657db4db89-j6v65"]
Oct 30 06:15:26.717320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:26.717267    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:27.283829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:27.283790    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c436f8-7a93-4543-952e-35149302152f" path="/var/lib/kubelet/pods/b3c436f8-7a93-4543-952e-35149302152f/volumes"
Oct 30 06:15:28.589605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:28.589576    5250 scope.go:122] "RemoveContainer" containerID="2543d8c52c73d3e6d3927f2be75b44fc6068e8d59d0a13d2685adf18ecf8f668"
Oct 30 06:15:28.600816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:28.600799    5250 scope.go:122] "RemoveContainer" containerID="d05170c3baf1a3221963be4f2f1ee7ee2b1d363e1ec369a1368a4a1fe87cfbcf"
Oct 30 06:15:28.622523 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:28.622498    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:28.913410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:28.913380    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"]
Oct 30 06:15:29.077902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:29.077869    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:29.223006 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:29.222905    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:29.366946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:29.366916    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:30.471308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:30.471268    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:32.375086 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:32.375048    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:15:32.592299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:32.592250    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:33.914171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:33.914119    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx" podUID="e6f19865-9040-40c3-aa79-93b1505c0224" containerName="payload-processing" containerID="cri-o://7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8" gracePeriod=25
Oct 30 06:15:34.168855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.168715    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:34.257799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.257775    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:15:34.375684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.375634    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f19865-9040-40c3-aa79-93b1505c0224-kube-api-access-4x92w\" (UniqueName: \"kubernetes.io/projected/e6f19865-9040-40c3-aa79-93b1505c0224-kube-api-access-4x92w\") pod \"e6f19865-9040-40c3-aa79-93b1505c0224\" (UID: \"e6f19865-9040-40c3-aa79-93b1505c0224\") "
Oct 30 06:15:34.375887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.375852    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f19865-9040-40c3-aa79-93b1505c0224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f19865-9040-40c3-aa79-93b1505c0224-plugins-config-volume\") pod \"e6f19865-9040-40c3-aa79-93b1505c0224\" (UID: \"e6f19865-9040-40c3-aa79-93b1505c0224\") "
Oct 30 06:15:34.376355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.376322    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f19865-9040-40c3-aa79-93b1505c0224-plugins-config-volume" pod "e6f19865-9040-40c3-aa79-93b1505c0224" (UID: "e6f19865-9040-40c3-aa79-93b1505c0224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:34.377227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.377191    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f19865-9040-40c3-aa79-93b1505c0224-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:34.378374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.378350    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f19865-9040-40c3-aa79-93b1505c0224-kube-api-access-4x92w" pod "e6f19865-9040-40c3-aa79-93b1505c0224" (UID: "e6f19865-9040-40c3-aa79-93b1505c0224"). InnerVolumeSpecName "kube-api-access-4x92w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:34.478946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.478869    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x92w\" (UniqueName: \"kubernetes.io/projected/e6f19865-9040-40c3-aa79-93b1505c0224-kube-api-access-4x92w\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:34.493371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.493326    5250 generic.go:356] "Generic (PLEG): container finished" podID="e6f19865-9040-40c3-aa79-93b1505c0224" containerID="7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8" exitCode=0
Oct 30 06:15:34.493371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.493360    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx" event={"ID":"e6f19865-9040-40c3-aa79-93b1505c0224","Type":"ContainerDied","Data":"7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8"}
Oct 30 06:15:34.493587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.493376    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"
Oct 30 06:15:34.493587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.493415    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx" event={"ID":"e6f19865-9040-40c3-aa79-93b1505c0224","Type":"ContainerDied","Data":"ddd721110274f265205d0c9623775121bddad0f8a511e9f62f298fecb4beb80a"}
Oct 30 06:15:34.493587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.493441    5250 scope.go:122] "RemoveContainer" containerID="7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8"
Oct 30 06:15:34.505327 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.505308    5250 scope.go:122] "RemoveContainer" containerID="7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8"
Oct 30 06:15:34.505609 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:34.505579    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8\": container with ID starting with 7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8 not found: ID does not exist" containerID="7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8"
Oct 30 06:15:34.505729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.505622    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8"} err="failed to get container status \"7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8\": rpc error: code = NotFound desc = could not find container \"7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8\": container with ID starting with 7ff64c71703fbb8d6bb5786d96da8bdfd2abd0ca6d55a4b959ebd59ee7a310a8 not found: ID does not exist"
Oct 30 06:15:34.542129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.542102    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"]
Oct 30 06:15:34.548487 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:34.548466    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-589d75488-4c2gx"]
Oct 30 06:15:35.283093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:35.283060    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f19865-9040-40c3-aa79-93b1505c0224" path="/var/lib/kubelet/pods/e6f19865-9040-40c3-aa79-93b1505c0224/volumes"
Oct 30 06:15:35.728921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:35.727560    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:36.400247 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:36.400213    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:37.588381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:37.588346    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:38.186398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:38.186353    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"]
Oct 30 06:15:38.243453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:38.243422    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:38.866535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:38.866504    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:39.123554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.123481    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"]
Oct 30 06:15:39.124217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124202    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124219    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124230    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124235    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124244    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124250    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124257    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124261    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124267    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:39.124272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124272    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124282    5250 cpu_manager.go:414] "containerMap: removing container" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124287    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124292    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124297    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124315    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124319    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124328    5250 cpu_manager.go:414] "containerMap: removing container" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124332    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124337    5250 cpu_manager.go:414] "containerMap: removing container" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124340    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124347    5250 cpu_manager.go:414] "containerMap: removing container" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124351    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124358    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124363    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124374    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124378    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124389    5250 cpu_manager.go:414] "containerMap: removing container" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124393    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124400    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124403    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124412    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e6f19865-9040-40c3-aa79-93b1505c0224" containerName="payload-processing"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124416    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f19865-9040-40c3-aa79-93b1505c0224" containerName="payload-processing"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124423    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124427    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124440    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124444    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124451    5250 cpu_manager.go:414] "containerMap: removing container" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124455    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124465    5250 cpu_manager.go:414] "containerMap: removing container" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124469    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124476    5250 cpu_manager.go:414] "containerMap: removing container" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124480    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124486    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124490    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124497    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124501    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124510    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124515    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124523    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b3c436f8-7a93-4543-952e-35149302152f" containerName="main"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124528    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c436f8-7a93-4543-952e-35149302152f" containerName="main"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124533    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124538    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124545    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124549    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124561    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124566    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124574    5250 cpu_manager.go:414] "containerMap: removing container" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124579    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124586    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:39.124569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124590    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124597    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124602    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124607    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124611    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124774    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124789    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124800    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124808    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124817    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124827    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124836    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124847    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124855    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124863    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124871    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124882    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124888    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124893    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124898    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124904    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124910    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124920    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124930    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124938    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124947    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f19865-9040-40c3-aa79-93b1505c0224" containerName="payload-processing"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124959    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124969    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124977    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124984    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.124994    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.125001    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c436f8-7a93-4543-952e-35149302152f" containerName="main"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.125008    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:39.126897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.125021    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:39.131065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.131034    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.151844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.149437    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"]
Oct 30 06:15:39.152384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.152355    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5a591d-serving-cert\""
Oct 30 06:15:39.232714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.232678    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:39.249848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.249807    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed3595f-719d-4076-abc8-f4ee6913c5ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.250031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.249869    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.250124 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.250099    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4d2j\" (UniqueName: \"kubernetes.io/projected/0ed3595f-719d-4076-abc8-f4ee6913c5ad-kube-api-access-m4d2j\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.250266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.250247    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.351755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.351710    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4d2j\" (UniqueName: \"kubernetes.io/projected/0ed3595f-719d-4076-abc8-f4ee6913c5ad-kube-api-access-m4d2j\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.351955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.351819    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.352071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.352052    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed3595f-719d-4076-abc8-f4ee6913c5ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.352130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.352096    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.353033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.353003    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed3595f-719d-4076-abc8-f4ee6913c5ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.355929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.355891    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.359826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.359648    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.403258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.382693    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:39.409204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.405902    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4d2j\" (UniqueName: \"kubernetes.io/projected/0ed3595f-719d-4076-abc8-f4ee6913c5ad-kube-api-access-m4d2j\") pod \"maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.489204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.488531    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:39.644657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.644586    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:39.927591 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:39.927561    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"]
Oct 30 06:15:39.928661 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:39.928592    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed3595f_719d_4076_abc8_f4ee6913c5ad.slice/crio-6dd968faeae0622192eb9053e236bf068794596a6a6857b5f3eaad0cf2bbb67a WatchSource:0}: Error finding container 6dd968faeae0622192eb9053e236bf068794596a6a6857b5f3eaad0cf2bbb67a: Status 404 returned error can't find the container with id 6dd968faeae0622192eb9053e236bf068794596a6a6857b5f3eaad0cf2bbb67a
Oct 30 06:15:40.587230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:40.585310    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6" event={"ID":"0ed3595f-719d-4076-abc8-f4ee6913c5ad","Type":"ContainerStarted","Data":"6dd968faeae0622192eb9053e236bf068794596a6a6857b5f3eaad0cf2bbb67a"}
Oct 30 06:15:41.176947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:41.176915    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:42.961829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:42.961752    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:43.219933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:43.219853    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:43.374984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:43.374948    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:43.613332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:43.613237    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6" event={"ID":"0ed3595f-719d-4076-abc8-f4ee6913c5ad","Type":"ContainerStarted","Data":"b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0"}
Oct 30 06:15:43.613519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:43.613391    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:43.652999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:43.652949    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6" podStartSLOduration=1.9188625 podStartE2EDuration="4.652935196s" totalImagesPullingTime="2.734072696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:43.651892794 +0000 UTC m=+2658.482930017" watchObservedRunningTime="2026-10-30 06:15:43.652935196 +0000 UTC m=+2658.483972413"
Oct 30 06:15:44.769384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.769354    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:44.919496 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.919457    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:44.921141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.921113    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"]
Oct 30 06:15:44.922288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922266    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:44.922345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922292    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:44.922345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922323    5250 cpu_manager.go:414] "containerMap: removing container" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:44.922345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922331    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:44.922443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922350    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:44.922443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922357    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:44.922443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922413    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:44.922443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922421    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:44.922443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922432    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:44.922443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922439    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab62622d-84ad-4903-9be4-3b6ec51291f3" containerName="cleanup"
Oct 30 06:15:44.922639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922457    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:44.922639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922464    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:44.922639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922478    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:44.922639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922485    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:44.922639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922498    5250 cpu_manager.go:414] "containerMap: removing container" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:44.922639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922504    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:44.922639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922517    5250 cpu_manager.go:414] "containerMap: removing container" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:44.922639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922523    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922720    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="404e0b49-889c-4102-91fd-b989dcd5445c" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922740    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc73291b-82f1-4261-bc0b-8712d8a49558" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922755    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff353edf-ad97-4352-a8b2-470fa965104c" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922767    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3d2145-c914-4668-b07d-c7058b2d3f59" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922778    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd00ce8-6874-4369-9022-1d1001fcbae3" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922789    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01694ac-ebec-4c70-a10a-39f4de159d25" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922799    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5783c3-b770-4ded-a301-7e607e09bdb4" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922812    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09445ae-5f09-4c18-a37e-3af42038cd1e" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922819    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ebfb55-80c6-4651-b46c-e588b914c30c" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922831    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3be9b16-06d1-4ab5-a8e8-1825afb98f39" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922843    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c63877-7c54-427d-b8bf-c0f1b22dcebb" containerName="cleanup"
Oct 30 06:15:44.922886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.922853    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="259369fd-d911-4b38-96d2-ee3b43d3c075" containerName="cleanup"
Oct 30 06:15:44.926794 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.926774    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:44.946055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:44.946031    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"]
Oct 30 06:15:45.030262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.010613    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fb9345-dockercfg-hg6g2\""
Oct 30 06:15:45.069249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.030593    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fb9345\""
Oct 30 06:15:45.069484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.055432    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f849b1-2e44-4485-bb6c-01c53cdaee28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6\" (UID: \"77f849b1-2e44-4485-bb6c-01c53cdaee28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:45.083203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.070055    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tw5v\" (UniqueName: \"kubernetes.io/projected/77f849b1-2e44-4485-bb6c-01c53cdaee28-kube-api-access-6tw5v\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6\" (UID: \"77f849b1-2e44-4485-bb6c-01c53cdaee28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:45.184932 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.184893    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tw5v\" (UniqueName: \"kubernetes.io/projected/77f849b1-2e44-4485-bb6c-01c53cdaee28-kube-api-access-6tw5v\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6\" (UID: \"77f849b1-2e44-4485-bb6c-01c53cdaee28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:45.185241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.185088    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f849b1-2e44-4485-bb6c-01c53cdaee28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6\" (UID: \"77f849b1-2e44-4485-bb6c-01c53cdaee28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:45.185859 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.185794    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f849b1-2e44-4485-bb6c-01c53cdaee28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6\" (UID: \"77f849b1-2e44-4485-bb6c-01c53cdaee28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:45.206590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.206566    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tw5v\" (UniqueName: \"kubernetes.io/projected/77f849b1-2e44-4485-bb6c-01c53cdaee28-kube-api-access-6tw5v\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6\" (UID: \"77f849b1-2e44-4485-bb6c-01c53cdaee28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:45.244244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.244211    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:45.555926 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:45.555892    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f849b1_2e44_4485_bb6c_01c53cdaee28.slice/crio-8666f803be6b82185846d2e3e92e9e231f6cbbe1b579e48785d229655cecc60c WatchSource:0}: Error finding container 8666f803be6b82185846d2e3e92e9e231f6cbbe1b579e48785d229655cecc60c: Status 404 returned error can't find the container with id 8666f803be6b82185846d2e3e92e9e231f6cbbe1b579e48785d229655cecc60c
Oct 30 06:15:45.556235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.556209    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"]
Oct 30 06:15:45.634236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:45.634205    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6" event={"ID":"77f849b1-2e44-4485-bb6c-01c53cdaee28","Type":"ContainerStarted","Data":"8666f803be6b82185846d2e3e92e9e231f6cbbe1b579e48785d229655cecc60c"}
Oct 30 06:15:46.644055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:46.644013    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6" event={"ID":"77f849b1-2e44-4485-bb6c-01c53cdaee28","Type":"ContainerStarted","Data":"4d44ca6687c4b857574250a6c519832f3a297a5985c7f7571a4635996d7d15d6"}
Oct 30 06:15:46.644721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:46.644091    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:46.681125 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:46.681072    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6" podStartSLOduration=2.681059192 podStartE2EDuration="2.681059192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:46.680423764 +0000 UTC m=+2661.511460985" watchObservedRunningTime="2026-10-30 06:15:46.681059192 +0000 UTC m=+2661.512096422"
Oct 30 06:15:46.757436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:46.757403    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:48.448876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:48.448840    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:49.628302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:49.628256    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:15:50.001722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:50.001688    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:52.572049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:52.572012    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:52.719634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:52.719595    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:53.073216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.073162    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:53.220876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.220830    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd" podUID="99671d1e-c801-4908-a9c0-2f68fbea8ed3" containerName="main" containerID="cri-o://113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659" gracePeriod=45
Oct 30 06:15:53.583729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.583646    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:15:53.665831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.665389    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tls-certs\" (UniqueName: \"kubernetes.io/secret/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tls-certs\") pod \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") "
Oct 30 06:15:53.666243 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.666212    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tmp-dir\") pod \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") "
Oct 30 06:15:53.666487 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.666460    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tmp-dir" pod "99671d1e-c801-4908-a9c0-2f68fbea8ed3" (UID: "99671d1e-c801-4908-a9c0-2f68fbea8ed3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:53.666682 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.666649    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99671d1e-c801-4908-a9c0-2f68fbea8ed3-kube-api-access-xrmm9\" (UniqueName: \"kubernetes.io/projected/99671d1e-c801-4908-a9c0-2f68fbea8ed3-kube-api-access-xrmm9\") pod \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") "
Oct 30 06:15:53.667054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.667034    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-model-cache\") pod \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") "
Oct 30 06:15:53.667562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.667534    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-dshm\") pod \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") "
Oct 30 06:15:53.668126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.667466    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-model-cache" pod "99671d1e-c801-4908-a9c0-2f68fbea8ed3" (UID: "99671d1e-c801-4908-a9c0-2f68fbea8ed3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:53.668353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.668323    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tls-certs" pod "99671d1e-c801-4908-a9c0-2f68fbea8ed3" (UID: "99671d1e-c801-4908-a9c0-2f68fbea8ed3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:15:53.668704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.668669    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-home" pod "99671d1e-c801-4908-a9c0-2f68fbea8ed3" (UID: "99671d1e-c801-4908-a9c0-2f68fbea8ed3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:53.668704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.668424    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-home\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-home\") pod \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\" (UID: \"99671d1e-c801-4908-a9c0-2f68fbea8ed3\") "
Oct 30 06:15:53.671344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.669426    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99671d1e-c801-4908-a9c0-2f68fbea8ed3-kube-api-access-xrmm9" pod "99671d1e-c801-4908-a9c0-2f68fbea8ed3" (UID: "99671d1e-c801-4908-a9c0-2f68fbea8ed3"). InnerVolumeSpecName "kube-api-access-xrmm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:53.671344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.669434    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:53.671344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.669487    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:53.671344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.669506    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:53.671344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.669521    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:53.691745 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.691713    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-dshm" pod "99671d1e-c801-4908-a9c0-2f68fbea8ed3" (UID: "99671d1e-c801-4908-a9c0-2f68fbea8ed3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:15:53.707480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.707447    5250 generic.go:356] "Generic (PLEG): container finished" podID="99671d1e-c801-4908-a9c0-2f68fbea8ed3" containerID="113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659" exitCode=0
Oct 30 06:15:53.707709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.707689    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"
Oct 30 06:15:53.707814 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.707725    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd" event={"ID":"99671d1e-c801-4908-a9c0-2f68fbea8ed3","Type":"ContainerDied","Data":"113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659"}
Oct 30 06:15:53.707814 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.707784    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd" event={"ID":"99671d1e-c801-4908-a9c0-2f68fbea8ed3","Type":"ContainerDied","Data":"f180eb074e824aada40f82a1943a6af6f08a5b344c2c5a143713c55c92bada44"}
Oct 30 06:15:53.707814 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.707809    5250 scope.go:122] "RemoveContainer" containerID="113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659"
Oct 30 06:15:53.722226 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.722205    5250 scope.go:122] "RemoveContainer" containerID="113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659"
Oct 30 06:15:53.722555 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:15:53.722529    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659\": container with ID starting with 113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659 not found: ID does not exist" containerID="113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659"
Oct 30 06:15:53.722670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.722565    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659"} err="failed to get container status \"113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659\": rpc error: code = NotFound desc = could not find container \"113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659\": container with ID starting with 113a8441d6a67d53998fecb8d5f42a7e521c26fc0db2a75376860e0f6bc5a659 not found: ID does not exist"
Oct 30 06:15:53.759320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.759284    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"]
Oct 30 06:15:53.763586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.763564    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-452a59/rate-test-model-452a59-kserve-758889db8f-hfwxd"]
Oct 30 06:15:53.770519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.770501    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99671d1e-c801-4908-a9c0-2f68fbea8ed3-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:53.770608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:53.770523    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmm9\" (UniqueName: \"kubernetes.io/projected/99671d1e-c801-4908-a9c0-2f68fbea8ed3-kube-api-access-xrmm9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:54.646353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.646318    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"]
Oct 30 06:15:54.647080 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.647064    5250 cpu_manager.go:414] "containerMap: removing container" podUID="99671d1e-c801-4908-a9c0-2f68fbea8ed3" containerName="main"
Oct 30 06:15:54.647128 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.647082    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99671d1e-c801-4908-a9c0-2f68fbea8ed3" containerName="main"
Oct 30 06:15:54.647260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.647245    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="99671d1e-c801-4908-a9c0-2f68fbea8ed3" containerName="main"
Oct 30 06:15:54.665225 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.665127    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.669311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.669279    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"]
Oct 30 06:15:54.678505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.678468    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:15:54.688791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.688765    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:15:54.697909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.697870    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5sb5\" (UniqueName: \"kubernetes.io/projected/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-kube-api-access-v5sb5\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.698261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.698239    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.698376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.698267    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.716335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.716298    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:54.799406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.799375    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.799406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.799408    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.799634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.799456    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5sb5\" (UniqueName: \"kubernetes.io/projected/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-kube-api-access-v5sb5\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.800232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.800128    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.802048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.802025    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.820986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.820960    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5sb5\" (UniqueName: \"kubernetes.io/projected/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-kube-api-access-v5sb5\") pod \"maas-api-revoke-sub-5c2869fa-q9m98\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:54.984205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:54.984115    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:55.283828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:55.283755    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99671d1e-c801-4908-a9c0-2f68fbea8ed3" path="/var/lib/kubelet/pods/99671d1e-c801-4908-a9c0-2f68fbea8ed3/volumes"
Oct 30 06:15:55.294211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:55.294172    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"]
Oct 30 06:15:55.296000 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:55.295975    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1651bb_7e96_4bd1_9ac4_39a0b5d0d1b2.slice/crio-dd4648042c90f30fcbd1f0f3562d155bc1ed0a7c71dbf0f6db71a581a19d6d15 WatchSource:0}: Error finding container dd4648042c90f30fcbd1f0f3562d155bc1ed0a7c71dbf0f6db71a581a19d6d15: Status 404 returned error can't find the container with id dd4648042c90f30fcbd1f0f3562d155bc1ed0a7c71dbf0f6db71a581a19d6d15
Oct 30 06:15:55.729639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:55.729599    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98" event={"ID":"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2","Type":"ContainerStarted","Data":"022e4e987551728719daa316fd372bdecce6adcf53bb88ec9b791802202bad50"}
Oct 30 06:15:55.730133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:55.729664    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98" event={"ID":"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2","Type":"ContainerStarted","Data":"dd4648042c90f30fcbd1f0f3562d155bc1ed0a7c71dbf0f6db71a581a19d6d15"}
Oct 30 06:15:55.780048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:55.779997    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98" podStartSLOduration=1.779981813 podStartE2EDuration="1.779981813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:55.776919404 +0000 UTC m=+2670.607956618" watchObservedRunningTime="2026-10-30 06:15:55.779981813 +0000 UTC m=+2670.611019030"
Oct 30 06:15:56.202054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.202004    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:56.738459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.738416    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8877-011f61"]
Oct 30 06:15:56.743491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.743464    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8877-011f61"
Oct 30 06:15:56.743977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.743954    5250 generic.go:356] "Generic (PLEG): container finished" podID="9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2" containerID="022e4e987551728719daa316fd372bdecce6adcf53bb88ec9b791802202bad50" exitCode=0
Oct 30 06:15:56.744169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.744047    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98" event={"ID":"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2","Type":"ContainerDied","Data":"022e4e987551728719daa316fd372bdecce6adcf53bb88ec9b791802202bad50"}
Oct 30 06:15:56.760270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.760246    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bnq\""
Oct 30 06:15:56.767784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.767751    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8877-011f61"]
Oct 30 06:15:56.813886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.813834    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:56.838238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.838167    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm78b\" (UniqueName: \"kubernetes.io/projected/d39909e4-17cb-4f02-b86f-ec094b44c753-kube-api-access-rm78b\") pod \"test-curl-8877-011f61\" (UID: \"d39909e4-17cb-4f02-b86f-ec094b44c753\") " pod="opendatahub/test-curl-8877-011f61"
Oct 30 06:15:56.940452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.940421    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm78b\" (UniqueName: \"kubernetes.io/projected/d39909e4-17cb-4f02-b86f-ec094b44c753-kube-api-access-rm78b\") pod \"test-curl-8877-011f61\" (UID: \"d39909e4-17cb-4f02-b86f-ec094b44c753\") " pod="opendatahub/test-curl-8877-011f61"
Oct 30 06:15:56.959717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.959668    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:56.962966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.962944    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm78b\" (UniqueName: \"kubernetes.io/projected/d39909e4-17cb-4f02-b86f-ec094b44c753-kube-api-access-rm78b\") pod \"test-curl-8877-011f61\" (UID: \"d39909e4-17cb-4f02-b86f-ec094b44c753\") " pod="opendatahub/test-curl-8877-011f61"
Oct 30 06:15:56.969894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.969863    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"]
Oct 30 06:15:56.977155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:56.977123    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b4ca14-7wjj2"]
Oct 30 06:15:57.065986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.063633    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8877-011f61"
Oct 30 06:15:57.098472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.098259    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:57.284021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.283991    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0f257f-bf55-40dc-81b1-af99c919362a" path="/var/lib/kubelet/pods/4a0f257f-bf55-40dc-81b1-af99c919362a/volumes"
Oct 30 06:15:57.397652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.397621    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8877-011f61"]
Oct 30 06:15:57.398795 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:57.398768    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd39909e4_17cb_4f02_b86f_ec094b44c753.slice/crio-d12a5022ddbe8f95f23ac500ebb2477d6ce9e222f1ccdcac364d4a1d630c4961 WatchSource:0}: Error finding container d12a5022ddbe8f95f23ac500ebb2477d6ce9e222f1ccdcac364d4a1d630c4961: Status 404 returned error can't find the container with id d12a5022ddbe8f95f23ac500ebb2477d6ce9e222f1ccdcac364d4a1d630c4961
Oct 30 06:15:57.470260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.470224    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:57.658170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.658096    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:15:57.756921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.756872    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8877-011f61" event={"ID":"d39909e4-17cb-4f02-b86f-ec094b44c753","Type":"ContainerStarted","Data":"787a013e6e12f0a6aa0d80f9f8953e07cce6f164234bed0c7a2a6afb8ccbf9af"}
Oct 30 06:15:57.757439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.756928    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8877-011f61" event={"ID":"d39909e4-17cb-4f02-b86f-ec094b44c753","Type":"ContainerStarted","Data":"d12a5022ddbe8f95f23ac500ebb2477d6ce9e222f1ccdcac364d4a1d630c4961"}
Oct 30 06:15:57.795527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.795466    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8877-011f61" podStartSLOduration=1.795446289 podStartE2EDuration="1.795446289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:57.791581926 +0000 UTC m=+2672.622619165" watchObservedRunningTime="2026-10-30 06:15:57.795446289 +0000 UTC m=+2672.626483507"
Oct 30 06:15:57.922506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:57.922478    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:58.057328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.057295    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-xks-ca\") pod \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") "
Oct 30 06:15:58.057631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.057613    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-kube-api-access-v5sb5\" (UniqueName: \"kubernetes.io/projected/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-kube-api-access-v5sb5\") pod \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") "
Oct 30 06:15:58.057738 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.057711    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-openshift-ca\") pod \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\" (UID: \"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2\") "
Oct 30 06:15:58.058083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.058046    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-openshift-ca" pod "9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2" (UID: "9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:15:58.058380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.058270    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:58.059629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.059608    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-xks-ca" pod "9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2" (UID: "9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:15:58.059773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.059760    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-kube-api-access-v5sb5" pod "9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2" (UID: "9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2"). InnerVolumeSpecName "kube-api-access-v5sb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:15:58.160218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.160137    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5sb5\" (UniqueName: \"kubernetes.io/projected/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-kube-api-access-v5sb5\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:58.160218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.160193    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:15:58.679374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.679339    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8877-011f61"]
Oct 30 06:15:58.686293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.686251    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8877-011f61"]
Oct 30 06:15:58.767369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.767336    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98" event={"ID":"9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2","Type":"ContainerDied","Data":"dd4648042c90f30fcbd1f0f3562d155bc1ed0a7c71dbf0f6db71a581a19d6d15"}
Oct 30 06:15:58.767369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.767363    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"
Oct 30 06:15:58.767868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.767395    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd4648042c90f30fcbd1f0f3562d155bc1ed0a7c71dbf0f6db71a581a19d6d15"
Oct 30 06:15:58.839734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.839706    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8877-513f47"]
Oct 30 06:15:58.840434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.840418    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d39909e4-17cb-4f02-b86f-ec094b44c753" containerName="test-curl-8877-011f61"
Oct 30 06:15:58.840490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.840436    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39909e4-17cb-4f02-b86f-ec094b44c753" containerName="test-curl-8877-011f61"
Oct 30 06:15:58.840490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.840467    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2" containerName="revoke-keys"
Oct 30 06:15:58.840490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.840481    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2" containerName="revoke-keys"
Oct 30 06:15:58.840607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.840598    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2" containerName="revoke-keys"
Oct 30 06:15:58.840643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.840616    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39909e4-17cb-4f02-b86f-ec094b44c753" containerName="test-curl-8877-011f61"
Oct 30 06:15:58.845885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.845868    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:15:58.853411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.853376    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:15:58.862848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.862821    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8877-513f47"]
Oct 30 06:15:58.975611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:58.975514    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hrj\" (UniqueName: \"kubernetes.io/projected/3becc85f-e48d-46b0-82ef-c5722695cab3-kube-api-access-47hrj\") pod \"test-curl-8877-513f47\" (UID: \"3becc85f-e48d-46b0-82ef-c5722695cab3\") " pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:15:59.077634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:59.077593    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hrj\" (UniqueName: \"kubernetes.io/projected/3becc85f-e48d-46b0-82ef-c5722695cab3-kube-api-access-47hrj\") pod \"test-curl-8877-513f47\" (UID: \"3becc85f-e48d-46b0-82ef-c5722695cab3\") " pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:15:59.100331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:59.100294    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hrj\" (UniqueName: \"kubernetes.io/projected/3becc85f-e48d-46b0-82ef-c5722695cab3-kube-api-access-47hrj\") pod \"test-curl-8877-513f47\" (UID: \"3becc85f-e48d-46b0-82ef-c5722695cab3\") " pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:15:59.161422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:59.161385    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:15:59.475158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:59.475129    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8877-513f47"]
Oct 30 06:15:59.475715 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:15:59.475689    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3becc85f_e48d_46b0_82ef_c5722695cab3.slice/crio-1fcff62ba64b807b1cfb5722e68781c92b28be0a1ed5cf4356a7ad504cd046ca WatchSource:0}: Error finding container 1fcff62ba64b807b1cfb5722e68781c92b28be0a1ed5cf4356a7ad504cd046ca: Status 404 returned error can't find the container with id 1fcff62ba64b807b1cfb5722e68781c92b28be0a1ed5cf4356a7ad504cd046ca
Oct 30 06:15:59.789132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:59.789029    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8877-513f47" event={"ID":"3becc85f-e48d-46b0-82ef-c5722695cab3","Type":"ContainerStarted","Data":"39e57a121d83c5a3fc8935bf1c661fe3cbb8fe1813b8ff760b4664b80f5b3fcd"}
Oct 30 06:15:59.789132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:59.789086    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8877-513f47" event={"ID":"3becc85f-e48d-46b0-82ef-c5722695cab3","Type":"ContainerStarted","Data":"1fcff62ba64b807b1cfb5722e68781c92b28be0a1ed5cf4356a7ad504cd046ca"}
Oct 30 06:15:59.789734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:59.789215    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8877-011f61" podUID="d39909e4-17cb-4f02-b86f-ec094b44c753" containerName="test-curl-8877-011f61" containerID="cri-o://787a013e6e12f0a6aa0d80f9f8953e07cce6f164234bed0c7a2a6afb8ccbf9af" gracePeriod=2
Oct 30 06:15:59.827593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:15:59.827549    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8877-513f47" podStartSLOduration=1.827536685 podStartE2EDuration="1.827536685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:59.82351459 +0000 UTC m=+2674.654551805" watchObservedRunningTime="2026-10-30 06:15:59.827536685 +0000 UTC m=+2674.658573901"
Oct 30 06:16:00.676288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:00.676255    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:00.689528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:00.689501    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8877-513f47"]
Oct 30 06:16:00.699711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:00.699682    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8877-513f47"]
Oct 30 06:16:01.242567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.242537    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"]
Oct 30 06:16:01.243329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.243313    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3becc85f-e48d-46b0-82ef-c5722695cab3" containerName="test-curl-8877-513f47"
Oct 30 06:16:01.243391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.243330    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3becc85f-e48d-46b0-82ef-c5722695cab3" containerName="test-curl-8877-513f47"
Oct 30 06:16:01.243471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.243461    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3becc85f-e48d-46b0-82ef-c5722695cab3" containerName="test-curl-8877-513f47"
Oct 30 06:16:01.247014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.246986    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.266088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.266060    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"]
Oct 30 06:16:01.270545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.270517    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:16:01.270861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.270831    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:16:01.415995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.415948    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzw66\" (UniqueName: \"kubernetes.io/projected/e8223c9b-9911-4274-b6a7-cb0161194e66-kube-api-access-kzw66\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.416276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.416251    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.416411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.416389    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.518584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.518507    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.518772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.518606    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.518772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.518744    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzw66\" (UniqueName: \"kubernetes.io/projected/e8223c9b-9911-4274-b6a7-cb0161194e66-kube-api-access-kzw66\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.519280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.519241    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.521039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.521015    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.540684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.540662    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzw66\" (UniqueName: \"kubernetes.io/projected/e8223c9b-9911-4274-b6a7-cb0161194e66-kube-api-access-kzw66\") pod \"maas-api-revoke-sub-09a9dac2-5skwd\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.562967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.562939    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:01.809936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.809907    5250 generic.go:356] "Generic (PLEG): container finished" podID="d39909e4-17cb-4f02-b86f-ec094b44c753" containerID="787a013e6e12f0a6aa0d80f9f8953e07cce6f164234bed0c7a2a6afb8ccbf9af" exitCode=137
Oct 30 06:16:01.810120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.810104    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8877-513f47" podUID="3becc85f-e48d-46b0-82ef-c5722695cab3" containerName="test-curl-8877-513f47" containerID="cri-o://39e57a121d83c5a3fc8935bf1c661fe3cbb8fe1813b8ff760b4664b80f5b3fcd" gracePeriod=2
Oct 30 06:16:01.875051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.875017    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"]
Oct 30 06:16:01.875871 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:16:01.875843    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8223c9b_9911_4274_b6a7_cb0161194e66.slice/crio-48bacb728bd3f376139060bb94a8fa43abc2fb241257f7704a5e0a509b924ba4 WatchSource:0}: Error finding container 48bacb728bd3f376139060bb94a8fa43abc2fb241257f7704a5e0a509b924ba4: Status 404 returned error can't find the container with id 48bacb728bd3f376139060bb94a8fa43abc2fb241257f7704a5e0a509b924ba4
Oct 30 06:16:01.959256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:01.959231    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8877-011f61"
Oct 30 06:16:02.128101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.128067    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39909e4-17cb-4f02-b86f-ec094b44c753-kube-api-access-rm78b\" (UniqueName: \"kubernetes.io/projected/d39909e4-17cb-4f02-b86f-ec094b44c753-kube-api-access-rm78b\") pod \"d39909e4-17cb-4f02-b86f-ec094b44c753\" (UID: \"d39909e4-17cb-4f02-b86f-ec094b44c753\") "
Oct 30 06:16:02.130474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.130445    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39909e4-17cb-4f02-b86f-ec094b44c753-kube-api-access-rm78b" pod "d39909e4-17cb-4f02-b86f-ec094b44c753" (UID: "d39909e4-17cb-4f02-b86f-ec094b44c753"). InnerVolumeSpecName "kube-api-access-rm78b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:02.230237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.230166    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm78b\" (UniqueName: \"kubernetes.io/projected/d39909e4-17cb-4f02-b86f-ec094b44c753-kube-api-access-rm78b\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:02.276100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.276068    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:02.376103 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:02.376018    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:16:02.421084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.421050    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"]
Oct 30 06:16:02.425728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.425703    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.451437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.451407    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-c2be21-openshift-default-dockercfg-cv24c\""
Oct 30 06:16:02.456961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.456930    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"]
Oct 30 06:16:02.534697 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.534654    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-credential-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.534855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.534783    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-podinfo\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.534855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.534837    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-envoy\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.535053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.535026    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cc351b-86a2-4fd7-9f17-9b831061c445-istiod-ca-cert\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.535145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.535090    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhct\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-kube-api-access-pfhct\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.535252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.535151    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-token\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.535312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.535255    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.535384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.535356    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-certs\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.535486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.535472    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-data\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.636843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.636808    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhct\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-kube-api-access-pfhct\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.636983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.636850    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-token\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.636983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.636892    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.636983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.636961    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-certs\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637056    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-data\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637085    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-credential-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637141    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-podinfo\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637199    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-envoy\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637476 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637245    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cc351b-86a2-4fd7-9f17-9b831061c445-istiod-ca-cert\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637476 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637424    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637498    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-certs\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637557    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-data\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637571    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-credential-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.637955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.637931    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cc351b-86a2-4fd7-9f17-9b831061c445-istiod-ca-cert\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.639707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.639685    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-envoy\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.639863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.639846    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-podinfo\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.659679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.659656    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-token\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.659959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.659937    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhct\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-kube-api-access-pfhct\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.742448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.742413    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:02.825222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.823549    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8877-011f61"
Oct 30 06:16:02.825222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.823557    5250 scope.go:122] "RemoveContainer" containerID="787a013e6e12f0a6aa0d80f9f8953e07cce6f164234bed0c7a2a6afb8ccbf9af"
Oct 30 06:16:02.828625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.827341    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd" event={"ID":"e8223c9b-9911-4274-b6a7-cb0161194e66","Type":"ContainerStarted","Data":"afbf04757896fb87df7fc182e53d250e62dfbad9f92126351c491e186a56443a"}
Oct 30 06:16:02.828625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.827404    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd" event={"ID":"e8223c9b-9911-4274-b6a7-cb0161194e66","Type":"ContainerStarted","Data":"48bacb728bd3f376139060bb94a8fa43abc2fb241257f7704a5e0a509b924ba4"}
Oct 30 06:16:02.865695 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:02.865644    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd" podStartSLOduration=1.865626549 podStartE2EDuration="1.865626549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:02.864880459 +0000 UTC m=+2677.695917674" watchObservedRunningTime="2026-10-30 06:16:02.865626549 +0000 UTC m=+2677.696663767"
Oct 30 06:16:02.876592 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:02.876563    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8877-011f61\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d39909e4-17cb-4f02-b86f-ec094b44c753" pod="opendatahub/test-curl-8877-011f61"
Oct 30 06:16:03.073642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.073608    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"]
Oct 30 06:16:03.073987 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:16:03.073965    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cc351b_86a2_4fd7_9f17_9b831061c445.slice/crio-3ccc2886a1a4721c370ba90190617b1041f67105d9f3d76e75a519068658a0dd WatchSource:0}: Error finding container 3ccc2886a1a4721c370ba90190617b1041f67105d9f3d76e75a519068658a0dd: Status 404 returned error can't find the container with id 3ccc2886a1a4721c370ba90190617b1041f67105d9f3d76e75a519068658a0dd
Oct 30 06:16:03.076880 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.076851    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:03.076961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.076915    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:03.076961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.076946    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:03.283616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.283579    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39909e4-17cb-4f02-b86f-ec094b44c753" path="/var/lib/kubelet/pods/d39909e4-17cb-4f02-b86f-ec094b44c753/volumes"
Oct 30 06:16:03.629249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.627871    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:03.781799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.781747    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:03.838416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.838392    5250 generic.go:356] "Generic (PLEG): container finished" podID="3becc85f-e48d-46b0-82ef-c5722695cab3" containerID="39e57a121d83c5a3fc8935bf1c661fe3cbb8fe1813b8ff760b4664b80f5b3fcd" exitCode=137
Oct 30 06:16:03.842617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.842587    5250 generic.go:356] "Generic (PLEG): container finished" podID="e8223c9b-9911-4274-b6a7-cb0161194e66" containerID="afbf04757896fb87df7fc182e53d250e62dfbad9f92126351c491e186a56443a" exitCode=0
Oct 30 06:16:03.842739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.842668    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd" event={"ID":"e8223c9b-9911-4274-b6a7-cb0161194e66","Type":"ContainerDied","Data":"afbf04757896fb87df7fc182e53d250e62dfbad9f92126351c491e186a56443a"}
Oct 30 06:16:03.844601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.844565    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" event={"ID":"46cc351b-86a2-4fd7-9f17-9b831061c445","Type":"ContainerStarted","Data":"8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b"}
Oct 30 06:16:03.844724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.844617    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" event={"ID":"46cc351b-86a2-4fd7-9f17-9b831061c445","Type":"ContainerStarted","Data":"3ccc2886a1a4721c370ba90190617b1041f67105d9f3d76e75a519068658a0dd"}
Oct 30 06:16:03.953301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.946143    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" podStartSLOduration=1.946125085 podStartE2EDuration="1.946125085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:03.942377428 +0000 UTC m=+2678.773414646" watchObservedRunningTime="2026-10-30 06:16:03.946125085 +0000 UTC m=+2678.777162304"
Oct 30 06:16:03.998435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:03.996434    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:04.278547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.249701    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:04.311600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.311572    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:16:04.360556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.360525    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3becc85f-e48d-46b0-82ef-c5722695cab3-kube-api-access-47hrj\" (UniqueName: \"kubernetes.io/projected/3becc85f-e48d-46b0-82ef-c5722695cab3-kube-api-access-47hrj\") pod \"3becc85f-e48d-46b0-82ef-c5722695cab3\" (UID: \"3becc85f-e48d-46b0-82ef-c5722695cab3\") "
Oct 30 06:16:04.386405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.386365    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3becc85f-e48d-46b0-82ef-c5722695cab3-kube-api-access-47hrj" pod "3becc85f-e48d-46b0-82ef-c5722695cab3" (UID: "3becc85f-e48d-46b0-82ef-c5722695cab3"). InnerVolumeSpecName "kube-api-access-47hrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:04.414047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.414011    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:04.475206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.463934    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47hrj\" (UniqueName: \"kubernetes.io/projected/3becc85f-e48d-46b0-82ef-c5722695cab3-kube-api-access-47hrj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:04.742719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.742683    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:04.744100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.744079    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 06:16:04.744238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.744125    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 06:16:04.854638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.854603    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:16:04.854820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:04.854655    5250 scope.go:122] "RemoveContainer" containerID="39e57a121d83c5a3fc8935bf1c661fe3cbb8fe1813b8ff760b4664b80f5b3fcd"
Oct 30 06:16:04.900517 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:04.900478    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8877-513f47\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="3becc85f-e48d-46b0-82ef-c5722695cab3" pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:16:05.099453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.099426    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:05.111146 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:05.111113    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8877-513f47\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="3becc85f-e48d-46b0-82ef-c5722695cab3" pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:16:05.190970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.190937    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-openshift-ca\") pod \"e8223c9b-9911-4274-b6a7-cb0161194e66\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") "
Oct 30 06:16:05.191158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.190986    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8223c9b-9911-4274-b6a7-cb0161194e66-kube-api-access-kzw66\" (UniqueName: \"kubernetes.io/projected/e8223c9b-9911-4274-b6a7-cb0161194e66-kube-api-access-kzw66\") pod \"e8223c9b-9911-4274-b6a7-cb0161194e66\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") "
Oct 30 06:16:05.191158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.191043    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-xks-ca\") pod \"e8223c9b-9911-4274-b6a7-cb0161194e66\" (UID: \"e8223c9b-9911-4274-b6a7-cb0161194e66\") "
Oct 30 06:16:05.191823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.191792    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-openshift-ca" pod "e8223c9b-9911-4274-b6a7-cb0161194e66" (UID: "e8223c9b-9911-4274-b6a7-cb0161194e66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:16:05.198986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.198962    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-xks-ca" pod "e8223c9b-9911-4274-b6a7-cb0161194e66" (UID: "e8223c9b-9911-4274-b6a7-cb0161194e66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:16:05.200571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.200544    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8223c9b-9911-4274-b6a7-cb0161194e66-kube-api-access-kzw66" pod "e8223c9b-9911-4274-b6a7-cb0161194e66" (UID: "e8223c9b-9911-4274-b6a7-cb0161194e66"). InnerVolumeSpecName "kube-api-access-kzw66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:05.290623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.290591    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3becc85f-e48d-46b0-82ef-c5722695cab3" path="/var/lib/kubelet/pods/3becc85f-e48d-46b0-82ef-c5722695cab3/volumes"
Oct 30 06:16:05.293262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.293232    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:05.293401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.293265    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzw66\" (UniqueName: \"kubernetes.io/projected/e8223c9b-9911-4274-b6a7-cb0161194e66-kube-api-access-kzw66\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:05.293401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.293284    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8223c9b-9911-4274-b6a7-cb0161194e66-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:05.299373 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:05.299341    5250 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8877-513f47\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="3becc85f-e48d-46b0-82ef-c5722695cab3" pod="opendatahub/test-curl-8877-513f47"
Oct 30 06:16:05.742961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.742918    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 06:16:05.743494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.742979    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 06:16:05.886839 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.886809    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"
Oct 30 06:16:05.887031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.886845    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd" event={"ID":"e8223c9b-9911-4274-b6a7-cb0161194e66","Type":"ContainerDied","Data":"48bacb728bd3f376139060bb94a8fa43abc2fb241257f7704a5e0a509b924ba4"}
Oct 30 06:16:05.887031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.886906    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48bacb728bd3f376139060bb94a8fa43abc2fb241257f7704a5e0a509b924ba4"
Oct 30 06:16:05.996261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:05.988750    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:06.743038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:06.743005    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 06:16:06.743536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:06.743061    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 06:16:07.742808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:07.742773    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 06:16:07.742973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:07.742841    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 06:16:07.811593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:07.811554    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:08.743559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:08.743434    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 06:16:08.743559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:08.743505    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 06:16:09.368234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:09.368170    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"]
Oct 30 06:16:09.512486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:09.512448    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:09.659517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:09.659473    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:09.877991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:09.877961    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:09.878872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:09.878843    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:09.879605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:09.879583    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:16:11.775452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:11.775417    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:13.511237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:13.511202    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:14.785024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:14.784991    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:14.951821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:14.951599    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:15.106341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:15.106239    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:15.263496 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:15.263460    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:15.516204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:15.501981    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:16.281947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:16.281904    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:17.624622 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:17.624588    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"]
Oct 30 06:16:17.717784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:17.717722    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:18.024799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:18.024711    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:19.700186 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:19.700141    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:22.298766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:22.298728    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:24.022443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.022408    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:24.391152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.391114    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" containerID="cri-o://8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9" gracePeriod=45
Oct 30 06:16:24.489560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.489515    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 30 06:16:24.613669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.613631    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"]
Oct 30 06:16:24.615136 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.615107    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e8223c9b-9911-4274-b6a7-cb0161194e66" containerName="revoke-keys"
Oct 30 06:16:24.615350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.615332    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8223c9b-9911-4274-b6a7-cb0161194e66" containerName="revoke-keys"
Oct 30 06:16:24.615655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.615634    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8223c9b-9911-4274-b6a7-cb0161194e66" containerName="revoke-keys"
Oct 30 06:16:24.621856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.620993    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.642440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.642371    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-2f1e4c-openshift-default-dockercfg-r26nk\""
Oct 30 06:16:24.642807 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.642778    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"]
Oct 30 06:16:24.743074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.743046    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 06:16:24.806361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806325    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.806557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806369    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.806557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806517    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.806712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806576    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.806712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806650    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.806832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806777    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.806832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806804    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.806964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806849    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.806964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.806931    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stcc4\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-kube-api-access-stcc4\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.908038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.907948    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-model-cache\") pod \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") "
Oct 30 06:16:24.908244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.908048    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-dshm\") pod \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") "
Oct 30 06:16:24.908352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.908283    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-model-cache" pod "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" (UID: "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:24.908352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.908296    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-home\") pod \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") "
Oct 30 06:16:24.908491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.908369    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-kube-api-access-9glk9\" (UniqueName: \"kubernetes.io/projected/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-kube-api-access-9glk9\") pod \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") "
Oct 30 06:16:24.908491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.908409    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tls-certs\") pod \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") "
Oct 30 06:16:24.909431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.908487    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-home" pod "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" (UID: "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:24.909431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.908778    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tmp-dir\") pod \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\" (UID: \"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109\") "
Oct 30 06:16:24.909431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.909080    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tmp-dir" pod "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" (UID: "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:24.909597 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.909467    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stcc4\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-kube-api-access-stcc4\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.909682 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.909663    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.909746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.909699    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.909801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.909765    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.909801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.909790    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.909912 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.909842    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.910009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.909992    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.910108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.910024    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.910108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.910075    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.910292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.910273    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:24.910365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.910297    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:24.910365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.910313    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:24.910586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.910563    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.911436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.911122    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.911436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.911323    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.911436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.911382    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.911712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.911571    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.912095 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.912068    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-dshm" pod "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" (UID: "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:24.912441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.912415    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-kube-api-access-9glk9" pod "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" (UID: "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109"). InnerVolumeSpecName "kube-api-access-9glk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:24.912757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.912735    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.913110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.913087    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tls-certs" pod "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" (UID: "a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:16:24.913987 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.913966    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.931353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.931328    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stcc4\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-kube-api-access-stcc4\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.931476 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.931423    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:24.981205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:24.961117    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:25.011448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.011420    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9glk9\" (UniqueName: \"kubernetes.io/projected/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-kube-api-access-9glk9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:25.011448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.011448    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:25.011612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.011458    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:25.116365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.116112    5250 generic.go:356] "Generic (PLEG): container finished" podID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerID="8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9" exitCode=0
Oct 30 06:16:25.116365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.116166    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" event={"ID":"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109","Type":"ContainerDied","Data":"8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9"}
Oct 30 06:16:25.116365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.116218    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w" event={"ID":"a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109","Type":"ContainerDied","Data":"b3b4d75c687734a4732b42cbbb57cfca76a2a5b7c234a5490cfbc429b26efdab"}
Oct 30 06:16:25.116365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.116241    5250 scope.go:122] "RemoveContainer" containerID="8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9"
Oct 30 06:16:25.116365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.116308    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"
Oct 30 06:16:25.145313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.144081    5250 scope.go:122] "RemoveContainer" containerID="8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9"
Oct 30 06:16:25.145313 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:25.144472    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9\": container with ID starting with 8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9 not found: ID does not exist" containerID="8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9"
Oct 30 06:16:25.145313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.144499    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9"} err="failed to get container status \"8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9\": rpc error: code = NotFound desc = could not find container \"8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9\": container with ID starting with 8c1561d960a0388e822a705ce667ad1aea4169fb1717e12281833c25833b84f9 not found: ID does not exist"
Oct 30 06:16:25.188485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.188386    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"]
Oct 30 06:16:25.195595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.195568    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/simulator-w4-698ab2-kserve-877d8b4b5-5w45w"]
Oct 30 06:16:25.291936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.291905    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" path="/var/lib/kubelet/pods/a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109/volumes"
Oct 30 06:16:25.313278 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.313251    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"]
Oct 30 06:16:25.313780 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:16:25.313752    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bec144_e4ab_4209_98a2_019bdfed2fe6.slice/crio-192a5b8d8e840c7a1d1c1c2d06c24bf4089456bd1680bf57337dfaa368a58b45 WatchSource:0}: Error finding container 192a5b8d8e840c7a1d1c1c2d06c24bf4089456bd1680bf57337dfaa368a58b45: Status 404 returned error can't find the container with id 192a5b8d8e840c7a1d1c1c2d06c24bf4089456bd1680bf57337dfaa368a58b45
Oct 30 06:16:25.316694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.316660    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:25.316766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.316729    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:25.316766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.316756    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:25.920324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.920295    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:16:25.985204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:25.984953    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:26.123005 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:26.122975    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:16:26.127163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:26.127129    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" event={"ID":"e6bec144-e4ab-4209-98a2-019bdfed2fe6","Type":"ContainerStarted","Data":"52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94"}
Oct 30 06:16:26.127389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:26.127263    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" event={"ID":"e6bec144-e4ab-4209-98a2-019bdfed2fe6","Type":"ContainerStarted","Data":"192a5b8d8e840c7a1d1c1c2d06c24bf4089456bd1680bf57337dfaa368a58b45"}
Oct 30 06:16:27.004209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:27.003856    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:27.010207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:27.009724    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 06:16:27.010207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:27.009786    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 06:16:27.712617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:27.712565    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:27.983262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:27.982373    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 06:16:27.983262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:27.982444    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 06:16:28.786344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:28.786309    5250 scope.go:122] "RemoveContainer" containerID="ea6a460fd2062c564b7c75342ae9c77226d67b0452f2921750fc32a9b23a8928"
Oct 30 06:16:28.974206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:28.969292    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 06:16:28.974206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:28.969357    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 06:16:29.299919 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:29.299872    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:29.976209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:29.975599    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 06:16:29.976209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:29.975703    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 06:16:30.740307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.740257    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" podStartSLOduration=6.740243876 podStartE2EDuration="6.740243876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:26.179507259 +0000 UTC m=+2701.010544488" watchObservedRunningTime="2026-10-30 06:16:30.740243876 +0000 UTC m=+2705.571281092"
Oct 30 06:16:30.746141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.746111    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"]
Oct 30 06:16:30.746897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.746881    5250 cpu_manager.go:414] "containerMap: removing container" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main"
Oct 30 06:16:30.746897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.746897    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main"
Oct 30 06:16:30.747028 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.747019    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f1f8bc-ea7d-49cd-8e2b-9ae9b9ced109" containerName="main"
Oct 30 06:16:30.752659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.752637    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.869200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.868571    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-de2bf2-openshift-default-dockercfg-s762q\""
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.880898    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.880983    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.881012    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llnbc\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-kube-api-access-llnbc\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.881088    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.881112    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b50a167-d13a-4729-9583-92075bb0fd64-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.881352    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.881399    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.881458    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b50a167-d13a-4729-9583-92075bb0fd64-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:30.915293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.881485    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.014111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.988658    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.022887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.022803    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llnbc\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-kube-api-access-llnbc\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.023277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.023256    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.023433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.023417    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b50a167-d13a-4729-9583-92075bb0fd64-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.023720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.023700    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.023879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.023859    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.024053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.024036    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b50a167-d13a-4729-9583-92075bb0fd64-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.025238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.025216    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.025646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.025621    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.025871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.024766    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.026263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.026242    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.027088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.027066    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b50a167-d13a-4729-9583-92075bb0fd64-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.027249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:30.994315    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 06:16:31.027405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.027381    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 06:16:31.028075 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.028052    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.028228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.008124    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.028562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.027276    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.033380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.033349    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b50a167-d13a-4729-9583-92075bb0fd64-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.033553 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.033370    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"]
Oct 30 06:16:31.041270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.041249    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.042382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.042356    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"]
Oct 30 06:16:31.047388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.046516    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llnbc\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-kube-api-access-llnbc\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.047762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.047723    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"]
Oct 30 06:16:31.052590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.052530    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.054215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.054158    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:16:31.083211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.054946    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:16:31.229499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.229467    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7qw\" (UniqueName: \"kubernetes.io/projected/59b2e5df-11ba-4d51-8201-041991e29895-kube-api-access-nr7qw\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.229689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.229558    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.229689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.229665    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.331611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.331518    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7qw\" (UniqueName: \"kubernetes.io/projected/59b2e5df-11ba-4d51-8201-041991e29895-kube-api-access-nr7qw\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.331807 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.331633    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.331807 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.331672    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.332372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.332348    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.333381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.333361    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:31.334373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.334351    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.383920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.383882    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7qw\" (UniqueName: \"kubernetes.io/projected/59b2e5df-11ba-4d51-8201-041991e29895-kube-api-access-nr7qw\") pod \"maas-api-revoke-sub-0dfd81e0-47t75\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.408581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.408542    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:31.488985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.488780    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:31.635427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.635397    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:31.715665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.715629    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"]
Oct 30 06:16:31.716404 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:16:31.716330    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b50a167_d13a_4729_9583_92075bb0fd64.slice/crio-f1858067a6f8ef231676a81922f9350183b3a5949690ba75eb8dcb69ad956e8a WatchSource:0}: Error finding container f1858067a6f8ef231676a81922f9350183b3a5949690ba75eb8dcb69ad956e8a: Status 404 returned error can't find the container with id f1858067a6f8ef231676a81922f9350183b3a5949690ba75eb8dcb69ad956e8a
Oct 30 06:16:31.720424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.720384    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:31.720535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.720478    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:31.720599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.720531    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:31.789881 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.789853    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"]
Oct 30 06:16:31.801993 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:16:31.801947    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b2e5df_11ba_4d51_8201_041991e29895.slice/crio-d4a036adc0ff3557adc3f00d351c92cc282bd5ac5f7eae48fedebea81dbb8131 WatchSource:0}: Error finding container d4a036adc0ff3557adc3f00d351c92cc282bd5ac5f7eae48fedebea81dbb8131: Status 404 returned error can't find the container with id d4a036adc0ff3557adc3f00d351c92cc282bd5ac5f7eae48fedebea81dbb8131
Oct 30 06:16:31.976330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.966342    5250 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 06:16:31.976330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:31.966403    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 06:16:32.225862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.225822    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75" event={"ID":"59b2e5df-11ba-4d51-8201-041991e29895","Type":"ContainerStarted","Data":"d4a036adc0ff3557adc3f00d351c92cc282bd5ac5f7eae48fedebea81dbb8131"}
Oct 30 06:16:32.229124 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.229041    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" event={"ID":"4b50a167-d13a-4729-9583-92075bb0fd64","Type":"ContainerStarted","Data":"d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1"}
Oct 30 06:16:32.229124 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.229110    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" event={"ID":"4b50a167-d13a-4729-9583-92075bb0fd64","Type":"ContainerStarted","Data":"f1858067a6f8ef231676a81922f9350183b3a5949690ba75eb8dcb69ad956e8a"}
Oct 30 06:16:32.287896 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.287717    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podStartSLOduration=2.287697551 podStartE2EDuration="2.287697551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:32.286488496 +0000 UTC m=+2707.117525714" watchObservedRunningTime="2026-10-30 06:16:32.287697551 +0000 UTC m=+2707.118734768"
Oct 30 06:16:32.333660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.333612    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:32.335901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.335553    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 06:16:32.335901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.335626    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 06:16:32.393572 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:32.393527    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:16:32.967680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.967645    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:32.968155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.968132    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:32.968792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:32.968770    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:16:33.013842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.013801    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw" podUID="4fa398e5-eadb-4eae-b34e-d3e36bf0d941" containerName="main" containerID="cri-o://14883aafdd0a14478b2ece2499e36d7858daf5bc53b1e315339f8c6864ef37a6" gracePeriod=45
Oct 30 06:16:33.231717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.231685    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"]
Oct 30 06:16:33.236725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.236698    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.243191 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.243151    5250 generic.go:356] "Generic (PLEG): container finished" podID="4fa398e5-eadb-4eae-b34e-d3e36bf0d941" containerID="14883aafdd0a14478b2ece2499e36d7858daf5bc53b1e315339f8c6864ef37a6" exitCode=0
Oct 30 06:16:33.243331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.243206    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw" event={"ID":"4fa398e5-eadb-4eae-b34e-d3e36bf0d941","Type":"ContainerDied","Data":"14883aafdd0a14478b2ece2499e36d7858daf5bc53b1e315339f8c6864ef37a6"}
Oct 30 06:16:33.245455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.245429    5250 generic.go:356] "Generic (PLEG): container finished" podID="59b2e5df-11ba-4d51-8201-041991e29895" containerID="c65c9f7fa73b174b3a0c2b011dc9098d30880ade92893084895fe6720ff71e9b" exitCode=0
Oct 30 06:16:33.245621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.245540    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75" event={"ID":"59b2e5df-11ba-4d51-8201-041991e29895","Type":"ContainerDied","Data":"c65c9f7fa73b174b3a0c2b011dc9098d30880ade92893084895fe6720ff71e9b"}
Oct 30 06:16:33.256551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.256531    5250 generic.go:356] "Generic (PLEG): container finished" podID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerID="8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b" exitCode=0
Oct 30 06:16:33.256683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.256636    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" event={"ID":"723e1a96-a612-4c85-9dbe-1d71d08938ba","Type":"ContainerDied","Data":"8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b"}
Oct 30 06:16:33.256758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.256694    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"]
Oct 30 06:16:33.257941 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.257862    5250 scope.go:122] "RemoveContainer" containerID="8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b"
Oct 30 06:16:33.262164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.261636    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:33.262164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.261721    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:33.262164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.261767    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:16:33.292773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.292747    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.296266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.296240    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbnw\" (UniqueName: \"kubernetes.io/projected/8d10798a-4aba-4df7-93d6-3763fb8df4f0-kube-api-access-mnbnw\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.297572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.297551    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.335087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.334652    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 06:16:33.335087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.334707    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 06:16:33.400100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.400070    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbnw\" (UniqueName: \"kubernetes.io/projected/8d10798a-4aba-4df7-93d6-3763fb8df4f0-kube-api-access-mnbnw\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.400500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.400472    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.400641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.400621    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.401958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.401902    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.405064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.404869    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.407152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.407133    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:16:33.422580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.422560    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbnw\" (UniqueName: \"kubernetes.io/projected/8d10798a-4aba-4df7-93d6-3763fb8df4f0-kube-api-access-mnbnw\") pod \"maas-api-revoke-sub-26b0bda6-6q2xc\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.501370    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-model-cache\") pod \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") "
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.501506    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-kube-api-access-jhrg7\" (UniqueName: \"kubernetes.io/projected/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-kube-api-access-jhrg7\") pod \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") "
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.501582    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-dshm\") pod \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") "
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.501631    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tmp-dir\") pod \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") "
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.501703    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tls-certs\") pod \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") "
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.501859    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-home\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-home\") pod \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\" (UID: \"4fa398e5-eadb-4eae-b34e-d3e36bf0d941\") "
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.503454    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-model-cache" pod "4fa398e5-eadb-4eae-b34e-d3e36bf0d941" (UID: "4fa398e5-eadb-4eae-b34e-d3e36bf0d941"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.504288    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tmp-dir" pod "4fa398e5-eadb-4eae-b34e-d3e36bf0d941" (UID: "4fa398e5-eadb-4eae-b34e-d3e36bf0d941"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:33.504617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.504378    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-home" pod "4fa398e5-eadb-4eae-b34e-d3e36bf0d941" (UID: "4fa398e5-eadb-4eae-b34e-d3e36bf0d941"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:33.507141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.507103    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-kube-api-access-jhrg7" pod "4fa398e5-eadb-4eae-b34e-d3e36bf0d941" (UID: "4fa398e5-eadb-4eae-b34e-d3e36bf0d941"). InnerVolumeSpecName "kube-api-access-jhrg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:33.511473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.511449    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-dshm" pod "4fa398e5-eadb-4eae-b34e-d3e36bf0d941" (UID: "4fa398e5-eadb-4eae-b34e-d3e36bf0d941"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:33.513484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.513453    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tls-certs" pod "4fa398e5-eadb-4eae-b34e-d3e36bf0d941" (UID: "4fa398e5-eadb-4eae-b34e-d3e36bf0d941"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:16:33.604162 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.604131    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:33.604162 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.604161    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:33.604162 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.604171    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:33.604453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.604197    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:33.604453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.604206    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:33.604453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.604214    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhrg7\" (UniqueName: \"kubernetes.io/projected/4fa398e5-eadb-4eae-b34e-d3e36bf0d941-kube-api-access-jhrg7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:33.654784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.654752    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:33.973212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:33.966205    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:34.019623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.019586    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"]
Oct 30 06:16:34.027055 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:16:34.027010    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d10798a_4aba_4df7_93d6_3763fb8df4f0.slice/crio-d1925f0fc078fd8d7be1d1766e19e6fc7080990940ada2f4a268f3e5523a908c WatchSource:0}: Error finding container d1925f0fc078fd8d7be1d1766e19e6fc7080990940ada2f4a268f3e5523a908c: Status 404 returned error can't find the container with id d1925f0fc078fd8d7be1d1766e19e6fc7080990940ada2f4a268f3e5523a908c
Oct 30 06:16:34.289303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.289220    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc" event={"ID":"8d10798a-4aba-4df7-93d6-3763fb8df4f0","Type":"ContainerStarted","Data":"d1925f0fc078fd8d7be1d1766e19e6fc7080990940ada2f4a268f3e5523a908c"}
Oct 30 06:16:34.291411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.291380    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" event={"ID":"723e1a96-a612-4c85-9dbe-1d71d08938ba","Type":"ContainerStarted","Data":"5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7"}
Oct 30 06:16:34.295909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.293802    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"
Oct 30 06:16:34.295909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.293831    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw" event={"ID":"4fa398e5-eadb-4eae-b34e-d3e36bf0d941","Type":"ContainerDied","Data":"2da267b3d3ea42a31fa591663a4c67f39be56d57c012283352bdffc70177fdf2"}
Oct 30 06:16:34.295909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.293882    5250 scope.go:122] "RemoveContainer" containerID="14883aafdd0a14478b2ece2499e36d7858daf5bc53b1e315339f8c6864ef37a6"
Oct 30 06:16:34.340784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.338358    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 06:16:34.340784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.338414    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 06:16:34.384325 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.383905    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:16:34.386559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.386138    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 06:16:34.386559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.386210    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 06:16:34.402516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.402445    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"]
Oct 30 06:16:34.408227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.408160    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-720e9c/auth-test-model-720e9c-kserve-7879fc7dd5-sjgqw"]
Oct 30 06:16:34.623660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.621608    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"]
Oct 30 06:16:34.624888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.624862    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"]
Oct 30 06:16:34.648010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.647984    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:34.738816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.735978    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-openshift-ca\") pod \"59b2e5df-11ba-4d51-8201-041991e29895\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") "
Oct 30 06:16:34.738816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.736125    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b2e5df-11ba-4d51-8201-041991e29895-kube-api-access-nr7qw\" (UniqueName: \"kubernetes.io/projected/59b2e5df-11ba-4d51-8201-041991e29895-kube-api-access-nr7qw\") pod \"59b2e5df-11ba-4d51-8201-041991e29895\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") "
Oct 30 06:16:34.738816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.736371    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-xks-ca\") pod \"59b2e5df-11ba-4d51-8201-041991e29895\" (UID: \"59b2e5df-11ba-4d51-8201-041991e29895\") "
Oct 30 06:16:34.738816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.737360    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-openshift-ca" pod "59b2e5df-11ba-4d51-8201-041991e29895" (UID: "59b2e5df-11ba-4d51-8201-041991e29895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:16:34.740976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.740362    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b2e5df-11ba-4d51-8201-041991e29895-kube-api-access-nr7qw" pod "59b2e5df-11ba-4d51-8201-041991e29895" (UID: "59b2e5df-11ba-4d51-8201-041991e29895"). InnerVolumeSpecName "kube-api-access-nr7qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:34.740976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.740469    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-xks-ca" pod "59b2e5df-11ba-4d51-8201-041991e29895" (UID: "59b2e5df-11ba-4d51-8201-041991e29895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:16:34.746384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.746355    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:34.840374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.839558    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:34.840374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.839601    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr7qw\" (UniqueName: \"kubernetes.io/projected/59b2e5df-11ba-4d51-8201-041991e29895-kube-api-access-nr7qw\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:34.840374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.839613    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b2e5df-11ba-4d51-8201-041991e29895-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:34.892169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:34.892007    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:35.083776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.083730    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:35.509204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.484932    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 06:16:35.509204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.486209    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 06:16:35.509204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.487304    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 06:16:35.509204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.487984    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 06:16:35.640017 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.639927    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa398e5-eadb-4eae-b34e-d3e36bf0d941" path="/var/lib/kubelet/pods/4fa398e5-eadb-4eae-b34e-d3e36bf0d941/volumes"
Oct 30 06:16:35.645305 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.645279    5250 generic.go:356] "Generic (PLEG): container finished" podID="8d10798a-4aba-4df7-93d6-3763fb8df4f0" containerID="385037203e921bbc752e3b63078b63cec50bd0d01a2d9eff0191883dcd9c87bf" exitCode=0
Oct 30 06:16:35.645456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.645344    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc" event={"ID":"8d10798a-4aba-4df7-93d6-3763fb8df4f0","Type":"ContainerDied","Data":"385037203e921bbc752e3b63078b63cec50bd0d01a2d9eff0191883dcd9c87bf"}
Oct 30 06:16:35.647465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.647437    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75" event={"ID":"59b2e5df-11ba-4d51-8201-041991e29895","Type":"ContainerDied","Data":"d4a036adc0ff3557adc3f00d351c92cc282bd5ac5f7eae48fedebea81dbb8131"}
Oct 30 06:16:35.647465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.647458    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"
Oct 30 06:16:35.647647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:35.647475    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4a036adc0ff3557adc3f00d351c92cc282bd5ac5f7eae48fedebea81dbb8131"
Oct 30 06:16:36.029257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:36.029224    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:16:36.333843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:36.333767    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 06:16:36.333995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:36.333843    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 06:16:36.388344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:36.388295    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 06:16:36.388789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:36.388759    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 06:16:36.514241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:36.514156    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:36.682221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:36.665607    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:37.017376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.017346    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:37.123458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.123428    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-openshift-ca\") pod \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") "
Oct 30 06:16:37.123643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.123553    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-xks-ca\") pod \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") "
Oct 30 06:16:37.123711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.123693    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d10798a-4aba-4df7-93d6-3763fb8df4f0-kube-api-access-mnbnw\" (UniqueName: \"kubernetes.io/projected/8d10798a-4aba-4df7-93d6-3763fb8df4f0-kube-api-access-mnbnw\") pod \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\" (UID: \"8d10798a-4aba-4df7-93d6-3763fb8df4f0\") "
Oct 30 06:16:37.124410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.124386    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-openshift-ca" pod "8d10798a-4aba-4df7-93d6-3763fb8df4f0" (UID: "8d10798a-4aba-4df7-93d6-3763fb8df4f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:16:37.126598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.126575    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d10798a-4aba-4df7-93d6-3763fb8df4f0-kube-api-access-mnbnw" pod "8d10798a-4aba-4df7-93d6-3763fb8df4f0" (UID: "8d10798a-4aba-4df7-93d6-3763fb8df4f0"). InnerVolumeSpecName "kube-api-access-mnbnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:37.129526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.129503    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-xks-ca" pod "8d10798a-4aba-4df7-93d6-3763fb8df4f0" (UID: "8d10798a-4aba-4df7-93d6-3763fb8df4f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:16:37.228248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.226279    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbnw\" (UniqueName: \"kubernetes.io/projected/8d10798a-4aba-4df7-93d6-3763fb8df4f0-kube-api-access-mnbnw\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:37.228248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.226315    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:37.228248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.226330    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d10798a-4aba-4df7-93d6-3763fb8df4f0-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:37.237612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.237560    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"]
Oct 30 06:16:37.343060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.341572    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 06:16:37.343060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.341648    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 06:16:37.402012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.401421    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 06:16:37.402012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.401476    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 06:16:37.416273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.416089    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:37.695907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.695868    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc" event={"ID":"8d10798a-4aba-4df7-93d6-3763fb8df4f0","Type":"ContainerDied","Data":"d1925f0fc078fd8d7be1d1766e19e6fc7080990940ada2f4a268f3e5523a908c"}
Oct 30 06:16:37.696576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.696554    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1925f0fc078fd8d7be1d1766e19e6fc7080990940ada2f4a268f3e5523a908c"
Oct 30 06:16:37.696750 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:37.696736    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"
Oct 30 06:16:38.049926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:38.049894    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:38.335697 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:38.335605    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 06:16:38.335697 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:38.335669    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 06:16:38.376205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:38.375263    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 06:16:38.376205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:38.375326    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 06:16:38.818514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:38.798161    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:39.082267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:39.081198    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:39.334301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:39.334215    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 06:16:39.334301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:39.334285    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 06:16:39.378204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:39.374578    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 06:16:39.378204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:39.374652    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 06:16:39.559212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:39.557043    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:39.626258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:39.626115    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7" podUID="e9b88e9a-ee31-4222-aff6-5be1b9d89223" containerName="payload-processing" containerID="cri-o://0fd89ee66aa98c3e8411b8d9f33f0586ab035f678a75bc1b825b87d7bf6709aa" gracePeriod=25
Oct 30 06:16:40.023195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.023003    5250 generic.go:356] "Generic (PLEG): container finished" podID="e9b88e9a-ee31-4222-aff6-5be1b9d89223" containerID="0fd89ee66aa98c3e8411b8d9f33f0586ab035f678a75bc1b825b87d7bf6709aa" exitCode=0
Oct 30 06:16:40.023195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.023047    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7" event={"ID":"e9b88e9a-ee31-4222-aff6-5be1b9d89223","Type":"ContainerDied","Data":"0fd89ee66aa98c3e8411b8d9f33f0586ab035f678a75bc1b825b87d7bf6709aa"}
Oct 30 06:16:40.023195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.023125    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7" event={"ID":"e9b88e9a-ee31-4222-aff6-5be1b9d89223","Type":"ContainerDied","Data":"bc43093e1f16016470c3b187f43c8b57b199d884ad2d11a8bedc9b23bac6315b"}
Oct 30 06:16:40.023195 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.023151    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc43093e1f16016470c3b187f43c8b57b199d884ad2d11a8bedc9b23bac6315b"
Oct 30 06:16:40.041417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.041392    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:16:40.196541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.196506    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9b88e9a-ee31-4222-aff6-5be1b9d89223-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b88e9a-ee31-4222-aff6-5be1b9d89223-plugins-config-volume\") pod \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\" (UID: \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\") "
Oct 30 06:16:40.196725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.196696    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b88e9a-ee31-4222-aff6-5be1b9d89223-kube-api-access-fbhlt\" (UniqueName: \"kubernetes.io/projected/e9b88e9a-ee31-4222-aff6-5be1b9d89223-kube-api-access-fbhlt\") pod \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\" (UID: \"e9b88e9a-ee31-4222-aff6-5be1b9d89223\") "
Oct 30 06:16:40.196910 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.196885    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b88e9a-ee31-4222-aff6-5be1b9d89223-plugins-config-volume" pod "e9b88e9a-ee31-4222-aff6-5be1b9d89223" (UID: "e9b88e9a-ee31-4222-aff6-5be1b9d89223"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:16:40.197334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.197315    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b88e9a-ee31-4222-aff6-5be1b9d89223-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:40.199063 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.199038    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b88e9a-ee31-4222-aff6-5be1b9d89223-kube-api-access-fbhlt" pod "e9b88e9a-ee31-4222-aff6-5be1b9d89223" (UID: "e9b88e9a-ee31-4222-aff6-5be1b9d89223"). InnerVolumeSpecName "kube-api-access-fbhlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:40.299126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.299027    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhlt\" (UniqueName: \"kubernetes.io/projected/e9b88e9a-ee31-4222-aff6-5be1b9d89223-kube-api-access-fbhlt\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:40.339152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.339119    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:40.339602 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.339580    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:40.340362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.340334    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:16:40.391271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.388440    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 06:16:40.391271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.388512    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 06:16:40.648514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.648468    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2" podUID="710b5db5-0169-4211-bc62-b9beb570471d" containerName="payload-pre-processing" containerID="cri-o://19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110" gracePeriod=25
Oct 30 06:16:40.992696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:40.992667    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:16:41.035632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.035595    5250 generic.go:356] "Generic (PLEG): container finished" podID="710b5db5-0169-4211-bc62-b9beb570471d" containerID="19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110" exitCode=0
Oct 30 06:16:41.036676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.036657    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"
Oct 30 06:16:41.036782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.036761    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2" event={"ID":"710b5db5-0169-4211-bc62-b9beb570471d","Type":"ContainerDied","Data":"19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110"}
Oct 30 06:16:41.036824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.036813    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2" event={"ID":"710b5db5-0169-4211-bc62-b9beb570471d","Type":"ContainerDied","Data":"a732bb57f0e105d9086f23b9b0607ad032272013102f2f2428c2ea803173c860"}
Oct 30 06:16:41.036866 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.036837    5250 scope.go:122] "RemoveContainer" containerID="19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110"
Oct 30 06:16:41.037013 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.036989    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"
Oct 30 06:16:41.062025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.062000    5250 scope.go:122] "RemoveContainer" containerID="19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110"
Oct 30 06:16:41.062482 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:41.062450    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110\": container with ID starting with 19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110 not found: ID does not exist" containerID="19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110"
Oct 30 06:16:41.062600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.062493    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110"} err="failed to get container status \"19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110\": rpc error: code = NotFound desc = could not find container \"19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110\": container with ID starting with 19165bf628fd78592444d5755ff20ac3948cc839c9f4fa6b79d7d1d434e85110 not found: ID does not exist"
Oct 30 06:16:41.106479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.106270    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"]
Oct 30 06:16:41.112307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.112276    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-89b469456-mlnw7"]
Oct 30 06:16:41.113815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.113795    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710b5db5-0169-4211-bc62-b9beb570471d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710b5db5-0169-4211-bc62-b9beb570471d-plugins-config-volume\") pod \"710b5db5-0169-4211-bc62-b9beb570471d\" (UID: \"710b5db5-0169-4211-bc62-b9beb570471d\") "
Oct 30 06:16:41.113901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.113827    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710b5db5-0169-4211-bc62-b9beb570471d-kube-api-access-bfjk5\" (UniqueName: \"kubernetes.io/projected/710b5db5-0169-4211-bc62-b9beb570471d-kube-api-access-bfjk5\") pod \"710b5db5-0169-4211-bc62-b9beb570471d\" (UID: \"710b5db5-0169-4211-bc62-b9beb570471d\") "
Oct 30 06:16:41.114329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.114300    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710b5db5-0169-4211-bc62-b9beb570471d-plugins-config-volume" pod "710b5db5-0169-4211-bc62-b9beb570471d" (UID: "710b5db5-0169-4211-bc62-b9beb570471d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:16:41.114704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.114683    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710b5db5-0169-4211-bc62-b9beb570471d-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:41.116418 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.116394    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710b5db5-0169-4211-bc62-b9beb570471d-kube-api-access-bfjk5" pod "710b5db5-0169-4211-bc62-b9beb570471d" (UID: "710b5db5-0169-4211-bc62-b9beb570471d"). InnerVolumeSpecName "kube-api-access-bfjk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:41.216579 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.216495    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfjk5\" (UniqueName: \"kubernetes.io/projected/710b5db5-0169-4211-bc62-b9beb570471d-kube-api-access-bfjk5\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:41.283585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.283548    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b88e9a-ee31-4222-aff6-5be1b9d89223" path="/var/lib/kubelet/pods/e9b88e9a-ee31-4222-aff6-5be1b9d89223/volumes"
Oct 30 06:16:41.378532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.378493    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"]
Oct 30 06:16:41.378870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.378854    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:16:41.380085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.380068    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:16:41.383449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.383426    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-5bc95cb76c-mkpj2"]
Oct 30 06:16:41.457172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:41.457143    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:42.734503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:42.734471    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:42.860234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:42.860199    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:43.283363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:43.283332    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710b5db5-0169-4211-bc62-b9beb570471d" path="/var/lib/kubelet/pods/710b5db5-0169-4211-bc62-b9beb570471d/volumes"
Oct 30 06:16:43.465196 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:43.465145    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:44.363792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.363741    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:44.451822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.451789    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"]
Oct 30 06:16:44.452606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452590    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4fa398e5-eadb-4eae-b34e-d3e36bf0d941" containerName="main"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452608    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa398e5-eadb-4eae-b34e-d3e36bf0d941" containerName="main"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452622    5250 cpu_manager.go:414] "containerMap: removing container" podUID="59b2e5df-11ba-4d51-8201-041991e29895" containerName="revoke-keys"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452627    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b2e5df-11ba-4d51-8201-041991e29895" containerName="revoke-keys"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452641    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e9b88e9a-ee31-4222-aff6-5be1b9d89223" containerName="payload-processing"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452646    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b88e9a-ee31-4222-aff6-5be1b9d89223" containerName="payload-processing"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452654    5250 cpu_manager.go:414] "containerMap: removing container" podUID="710b5db5-0169-4211-bc62-b9beb570471d" containerName="payload-pre-processing"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452658    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710b5db5-0169-4211-bc62-b9beb570471d" containerName="payload-pre-processing"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452666    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8d10798a-4aba-4df7-93d6-3763fb8df4f0" containerName="revoke-keys"
Oct 30 06:16:44.452678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452670    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d10798a-4aba-4df7-93d6-3763fb8df4f0" containerName="revoke-keys"
Oct 30 06:16:44.452973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452806    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa398e5-eadb-4eae-b34e-d3e36bf0d941" containerName="main"
Oct 30 06:16:44.452973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452813    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b2e5df-11ba-4d51-8201-041991e29895" containerName="revoke-keys"
Oct 30 06:16:44.452973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452822    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b88e9a-ee31-4222-aff6-5be1b9d89223" containerName="payload-processing"
Oct 30 06:16:44.452973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452827    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="710b5db5-0169-4211-bc62-b9beb570471d" containerName="payload-pre-processing"
Oct 30 06:16:44.452973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.452834    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d10798a-4aba-4df7-93d6-3763fb8df4f0" containerName="revoke-keys"
Oct 30 06:16:44.455977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.455959    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:44.475831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.475801    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"]
Oct 30 06:16:44.479560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.479533    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15b7328-9cdf-413c-ba5e-cd1f6821687a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns\" (UID: \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:44.479729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.479689    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dpkh\" (UniqueName: \"kubernetes.io/projected/e15b7328-9cdf-413c-ba5e-cd1f6821687a-kube-api-access-7dpkh\") pod \"payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns\" (UID: \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:44.480039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.480011    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07b278-dockercfg-92k4m\""
Oct 30 06:16:44.480039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.480012    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-07b278\""
Oct 30 06:16:44.512570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.512531    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:44.692001 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.690946    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dpkh\" (UniqueName: \"kubernetes.io/projected/e15b7328-9cdf-413c-ba5e-cd1f6821687a-kube-api-access-7dpkh\") pod \"payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns\" (UID: \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:44.692001 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.691325    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15b7328-9cdf-413c-ba5e-cd1f6821687a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns\" (UID: \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:44.693204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.692314    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15b7328-9cdf-413c-ba5e-cd1f6821687a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns\" (UID: \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:44.741979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.740554    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dpkh\" (UniqueName: \"kubernetes.io/projected/e15b7328-9cdf-413c-ba5e-cd1f6821687a-kube-api-access-7dpkh\") pod \"payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns\" (UID: \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:44.780203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:44.776479    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:45.100691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:45.100662    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"]
Oct 30 06:16:45.101831 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:16:45.101808    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15b7328_9cdf_413c_ba5e_cd1f6821687a.slice/crio-dd4a87920d89b1a394c911afcf65a25be45643582b98da3b67e6a74a937005da WatchSource:0}: Error finding container dd4a87920d89b1a394c911afcf65a25be45643582b98da3b67e6a74a937005da: Status 404 returned error can't find the container with id dd4a87920d89b1a394c911afcf65a25be45643582b98da3b67e6a74a937005da
Oct 30 06:16:46.038873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:46.038107    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"]
Oct 30 06:16:46.050215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:46.047347    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:46.063203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:46.057567    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79128713-7ntq5"]
Oct 30 06:16:46.110876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:46.110835    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns" event={"ID":"e15b7328-9cdf-413c-ba5e-cd1f6821687a","Type":"ContainerStarted","Data":"f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014"}
Oct 30 06:16:46.111093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:46.110978    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:46.111093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:46.111009    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns" event={"ID":"e15b7328-9cdf-413c-ba5e-cd1f6821687a","Type":"ContainerStarted","Data":"dd4a87920d89b1a394c911afcf65a25be45643582b98da3b67e6a74a937005da"}
Oct 30 06:16:46.148429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:46.148374    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns" podStartSLOduration=2.148357038 podStartE2EDuration="2.148357038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:46.147537874 +0000 UTC m=+2720.978575092" watchObservedRunningTime="2026-10-30 06:16:46.148357038 +0000 UTC m=+2720.979394256"
Oct 30 06:16:46.940166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:46.934342    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:47.096208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:47.095209    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:47.283218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:47.283081    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66db160-1292-4851-b8cc-4e53182b0acd" path="/var/lib/kubelet/pods/b66db160-1292-4851-b8cc-4e53182b0acd/volumes"
Oct 30 06:16:47.770262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:47.763384    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:48.402585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:48.402547    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:50.230855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:50.229914    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:51.965798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:51.965766    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:52.232993 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.232913    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"]
Oct 30 06:16:52.404822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.404790    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:52.408534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.408504    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c" podUID="b60419d0-842b-4dd4-bdcd-df58427cf9d7" containerName="main" containerID="cri-o://d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955" gracePeriod=45
Oct 30 06:16:52.463389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.463346    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c" podUID="b60419d0-842b-4dd4-bdcd-df58427cf9d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.221:8000/ready\": dial tcp 10.133.0.221:8000: connect: connection refused"
Oct 30 06:16:52.635849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.635740    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:52.714641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.714615    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:16:52.763700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.763666    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-dshm\") pod \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") "
Oct 30 06:16:52.763921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.763800    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-model-cache\") pod \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") "
Oct 30 06:16:52.763921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.763848    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tmp-dir\") pod \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") "
Oct 30 06:16:52.764120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.764056    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-home\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-home\") pod \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") "
Oct 30 06:16:52.764120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.764112    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tls-certs\") pod \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") "
Oct 30 06:16:52.764365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.764223    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60419d0-842b-4dd4-bdcd-df58427cf9d7-kube-api-access-x85ns\" (UniqueName: \"kubernetes.io/projected/b60419d0-842b-4dd4-bdcd-df58427cf9d7-kube-api-access-x85ns\") pod \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\" (UID: \"b60419d0-842b-4dd4-bdcd-df58427cf9d7\") "
Oct 30 06:16:52.764365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.764267    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tmp-dir" pod "b60419d0-842b-4dd4-bdcd-df58427cf9d7" (UID: "b60419d0-842b-4dd4-bdcd-df58427cf9d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:52.764544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.764463    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-home" pod "b60419d0-842b-4dd4-bdcd-df58427cf9d7" (UID: "b60419d0-842b-4dd4-bdcd-df58427cf9d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:52.765227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.765202    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:52.765349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.765231    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:52.765349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.765309    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-model-cache" pod "b60419d0-842b-4dd4-bdcd-df58427cf9d7" (UID: "b60419d0-842b-4dd4-bdcd-df58427cf9d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:52.776726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.776656    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-dshm" pod "b60419d0-842b-4dd4-bdcd-df58427cf9d7" (UID: "b60419d0-842b-4dd4-bdcd-df58427cf9d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:16:52.776847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.776762    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60419d0-842b-4dd4-bdcd-df58427cf9d7-kube-api-access-x85ns" pod "b60419d0-842b-4dd4-bdcd-df58427cf9d7" (UID: "b60419d0-842b-4dd4-bdcd-df58427cf9d7"). InnerVolumeSpecName "kube-api-access-x85ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:52.776847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.776815    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tls-certs" pod "b60419d0-842b-4dd4-bdcd-df58427cf9d7" (UID: "b60419d0-842b-4dd4-bdcd-df58427cf9d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:16:52.894205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.888661    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60419d0-842b-4dd4-bdcd-df58427cf9d7-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:52.894205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.888699    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x85ns\" (UniqueName: \"kubernetes.io/projected/b60419d0-842b-4dd4-bdcd-df58427cf9d7-kube-api-access-x85ns\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:52.894205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.888713    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:52.894205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:52.888726    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60419d0-842b-4dd4-bdcd-df58427cf9d7-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:53.267383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.267352    5250 generic.go:356] "Generic (PLEG): container finished" podID="b60419d0-842b-4dd4-bdcd-df58427cf9d7" containerID="d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955" exitCode=0
Oct 30 06:16:53.267829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.267396    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"
Oct 30 06:16:53.267829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.267402    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c" event={"ID":"b60419d0-842b-4dd4-bdcd-df58427cf9d7","Type":"ContainerDied","Data":"d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955"}
Oct 30 06:16:53.267829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.267432    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c" event={"ID":"b60419d0-842b-4dd4-bdcd-df58427cf9d7","Type":"ContainerDied","Data":"a38ebc595d737f27cfadcfc59ce6f8f605cf945a02355543dc1cf5798510498e"}
Oct 30 06:16:53.267829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.267452    5250 scope.go:122] "RemoveContainer" containerID="d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955"
Oct 30 06:16:53.287104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.287083    5250 scope.go:122] "RemoveContainer" containerID="d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955"
Oct 30 06:16:53.287426 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:53.287404    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955\": container with ID starting with d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955 not found: ID does not exist" containerID="d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955"
Oct 30 06:16:53.287512 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.287436    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955"} err="failed to get container status \"d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955\": rpc error: code = NotFound desc = could not find container \"d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955\": container with ID starting with d12e5d05f4cd46591de76ca5efbb040f74cd7922210c06bf6afa2fba42876955 not found: ID does not exist"
Oct 30 06:16:53.325483 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.325447    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"]
Oct 30 06:16:53.331496 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.331455    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95498c/auth-test-model-95498c-kserve-d46cbc545-tfx9c"]
Oct 30 06:16:53.773021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:53.772987    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:54.248820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:54.248784    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:55.283675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:55.283637    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60419d0-842b-4dd4-bdcd-df58427cf9d7" path="/var/lib/kubelet/pods/b60419d0-842b-4dd4-bdcd-df58427cf9d7/volumes"
Oct 30 06:16:55.376811 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:55.376777    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:55.655437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:55.655394    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"]
Oct 30 06:16:55.732084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:55.732052    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:57.006624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.006591    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:57.214110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.214082    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:16:57.233639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.233602    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f" podUID="8009548a-1020-4ebb-bb28-a363a1b6f9ee" containerName="payload-processing" containerID="cri-o://f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a" gracePeriod=25
Oct 30 06:16:57.536151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.536125    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:16:57.675024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.674992    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8009548a-1020-4ebb-bb28-a363a1b6f9ee-kube-api-access-gflc5\" (UniqueName: \"kubernetes.io/projected/8009548a-1020-4ebb-bb28-a363a1b6f9ee-kube-api-access-gflc5\") pod \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\" (UID: \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\") "
Oct 30 06:16:57.675242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.675127    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8009548a-1020-4ebb-bb28-a363a1b6f9ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8009548a-1020-4ebb-bb28-a363a1b6f9ee-plugins-config-volume\") pod \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\" (UID: \"8009548a-1020-4ebb-bb28-a363a1b6f9ee\") "
Oct 30 06:16:57.676149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.676103    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8009548a-1020-4ebb-bb28-a363a1b6f9ee-plugins-config-volume" pod "8009548a-1020-4ebb-bb28-a363a1b6f9ee" (UID: "8009548a-1020-4ebb-bb28-a363a1b6f9ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:16:57.687214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.677539    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8009548a-1020-4ebb-bb28-a363a1b6f9ee-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:57.687214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.677842    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8009548a-1020-4ebb-bb28-a363a1b6f9ee-kube-api-access-gflc5" pod "8009548a-1020-4ebb-bb28-a363a1b6f9ee" (UID: "8009548a-1020-4ebb-bb28-a363a1b6f9ee"). InnerVolumeSpecName "kube-api-access-gflc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:16:57.778655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:57.778621    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gflc5\" (UniqueName: \"kubernetes.io/projected/8009548a-1020-4ebb-bb28-a363a1b6f9ee-kube-api-access-gflc5\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:16:58.035310 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.035229    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:58.180311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.180268    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:58.317037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.316952    5250 generic.go:356] "Generic (PLEG): container finished" podID="8009548a-1020-4ebb-bb28-a363a1b6f9ee" containerID="f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a" exitCode=0
Oct 30 06:16:58.317037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.317009    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"
Oct 30 06:16:58.317337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.317045    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f" event={"ID":"8009548a-1020-4ebb-bb28-a363a1b6f9ee","Type":"ContainerDied","Data":"f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a"}
Oct 30 06:16:58.317337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.317105    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f" event={"ID":"8009548a-1020-4ebb-bb28-a363a1b6f9ee","Type":"ContainerDied","Data":"22356e674f1cd400836617464aa629bca3f5b253dd5cdf4474636eee090b332c"}
Oct 30 06:16:58.317337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.317136    5250 scope.go:122] "RemoveContainer" containerID="f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a"
Oct 30 06:16:58.329433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.329412    5250 scope.go:122] "RemoveContainer" containerID="f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a"
Oct 30 06:16:58.329757 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:16:58.329727    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a\": container with ID starting with f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a not found: ID does not exist" containerID="f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a"
Oct 30 06:16:58.329872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.329783    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a"} err="failed to get container status \"f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a\": rpc error: code = NotFound desc = could not find container \"f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a\": container with ID starting with f2d5d0749daef161b07aca4efa34f9cdeb5a5bd59f07e77ed78f693cf84a711a not found: ID does not exist"
Oct 30 06:16:58.365311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.365287    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"]
Oct 30 06:16:58.375395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.375372    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-87d949684-nst8f"]
Oct 30 06:16:58.548614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:58.548582    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:16:59.284371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:16:59.284331    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8009548a-1020-4ebb-bb28-a363a1b6f9ee" path="/var/lib/kubelet/pods/8009548a-1020-4ebb-bb28-a363a1b6f9ee/volumes"
Oct 30 06:17:00.057308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:00.057272    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:01.740630 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:01.740595    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:02.377829 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:17:02.377794    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:17:03.028106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:03.028063    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:03.172621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:03.172589    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:03.865441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:03.865404    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:05.453542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:05.453510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:07.530513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:07.530434    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:09.057515 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:09.057364    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:09.230446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:09.230408    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:09.389193 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:09.389153    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:10.632837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:10.632798    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:10.696765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:10.696693    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75" podUID="fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" containerName="main" containerID="cri-o://791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a" gracePeriod=45
Oct 30 06:17:10.998091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:10.998025    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:17:11.083490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.083457    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-model-cache\") pod \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") "
Oct 30 06:17:11.083698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.083513    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-kube-api-access-lr7px\" (UniqueName: \"kubernetes.io/projected/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-kube-api-access-lr7px\") pod \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") "
Oct 30 06:17:11.083698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.083606    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tmp-dir\") pod \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") "
Oct 30 06:17:11.083698 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.083672    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-dshm\") pod \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") "
Oct 30 06:17:11.083881 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.083767    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-model-cache" pod "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" (UID: "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:11.083881 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.083775    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tls-certs\") pod \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") "
Oct 30 06:17:11.084056 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.083896    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-home\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-home\") pod \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\" (UID: \"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc\") "
Oct 30 06:17:11.084250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.084221    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tmp-dir" pod "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" (UID: "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:11.084646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.084621    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:11.084747 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.084653    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:11.084984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.084953    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-home" pod "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" (UID: "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:11.086339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.086312    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-dshm" pod "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" (UID: "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:11.086339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.086323    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tls-certs" pod "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" (UID: "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:17:11.086339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.086330    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-kube-api-access-lr7px" pod "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" (UID: "fd76cc8a-8336-4d83-a616-f53b4a6fd4dc"). InnerVolumeSpecName "kube-api-access-lr7px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:17:11.186671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.186641    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr7px\" (UniqueName: \"kubernetes.io/projected/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-kube-api-access-lr7px\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:11.186671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.186671    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:11.186847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.186682    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:11.186847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.186691    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:11.429990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.429949    5250 generic.go:356] "Generic (PLEG): container finished" podID="fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" containerID="791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a" exitCode=0
Oct 30 06:17:11.430170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.429995    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"
Oct 30 06:17:11.430170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.430035    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75" event={"ID":"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc","Type":"ContainerDied","Data":"791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a"}
Oct 30 06:17:11.430170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.430078    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75" event={"ID":"fd76cc8a-8336-4d83-a616-f53b4a6fd4dc","Type":"ContainerDied","Data":"72d82ade8801ec3438fd8d99a7080b43c5e8e85729206a52fa857a1817026c32"}
Oct 30 06:17:11.430170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.430094    5250 scope.go:122] "RemoveContainer" containerID="791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a"
Oct 30 06:17:11.444956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.444937    5250 scope.go:122] "RemoveContainer" containerID="791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a"
Oct 30 06:17:11.445279 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:17:11.445251    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a\": container with ID starting with 791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a not found: ID does not exist" containerID="791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a"
Oct 30 06:17:11.445341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.445294    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a"} err="failed to get container status \"791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a\": rpc error: code = NotFound desc = could not find container \"791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a\": container with ID starting with 791798b6396c0fca31ae7b8661275c1f2bfc89f39a85d7ee9689580458685b5a not found: ID does not exist"
Oct 30 06:17:11.478846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.478819    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"]
Oct 30 06:17:11.489005 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:11.488982    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-c2be21/sub-test-model-c2be21-kserve-66d6988775-zkp75"]
Oct 30 06:17:12.700613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:12.700571    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:13.282970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:13.282939    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" path="/var/lib/kubelet/pods/fd76cc8a-8336-4d83-a616-f53b4a6fd4dc/volumes"
Oct 30 06:17:15.909885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:15.909828    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:16.887915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:16.887868    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:18.598872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:18.598837    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:19.577758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:19.577720    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"]
Oct 30 06:17:19.652233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:19.652162    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:20.348173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.348117    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:20.488873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.488844    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:20.662173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.662137    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"]
Oct 30 06:17:20.663389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663359    5250 cpu_manager.go:414] "containerMap: removing container" podUID="fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" containerName="main"
Oct 30 06:17:20.663389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663389    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" containerName="main"
Oct 30 06:17:20.663601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663406    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b60419d0-842b-4dd4-bdcd-df58427cf9d7" containerName="main"
Oct 30 06:17:20.663601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663414    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60419d0-842b-4dd4-bdcd-df58427cf9d7" containerName="main"
Oct 30 06:17:20.663601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663483    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8009548a-1020-4ebb-bb28-a363a1b6f9ee" containerName="payload-processing"
Oct 30 06:17:20.663601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663492    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8009548a-1020-4ebb-bb28-a363a1b6f9ee" containerName="payload-processing"
Oct 30 06:17:20.663778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663710    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8009548a-1020-4ebb-bb28-a363a1b6f9ee" containerName="payload-processing"
Oct 30 06:17:20.663778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663741    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd76cc8a-8336-4d83-a616-f53b4a6fd4dc" containerName="main"
Oct 30 06:17:20.663778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.663751    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60419d0-842b-4dd4-bdcd-df58427cf9d7" containerName="main"
Oct 30 06:17:20.669942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.669915    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.688164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.688132    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-istio-token\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.688391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.688363    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-certs\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.688500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.688474    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-data\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.690303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.690266    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b79e875-5537-492e-aa94-3a015701a0f8-istio-podinfo\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.690583 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.690557    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b79e875-5537-492e-aa94-3a015701a0f8-istiod-ca-cert\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.690844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.690823    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-credential-socket\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.691120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.691099    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-socket\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.691370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.691349    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"]
Oct 30 06:17:20.691497 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.691476    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkq47\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-kube-api-access-fkq47\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.691634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.691615    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-envoy\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.694869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.694841    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7fdb95-openshift-default-dockercfg-snd5d\""
Oct 30 06:17:20.749516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.749460    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:20.793559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793518    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b79e875-5537-492e-aa94-3a015701a0f8-istio-podinfo\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.793742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793567    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b79e875-5537-492e-aa94-3a015701a0f8-istiod-ca-cert\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.793742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793605    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-credential-socket\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.793742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793674    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-socket\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.793742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793707    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkq47\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-kube-api-access-fkq47\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.793742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793741    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-envoy\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.794037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793821    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-istio-token\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.794037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793940    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-certs\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.794037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.793988    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-data\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.794275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.794234    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-socket\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.794383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.794358    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-data\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.794522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.794493    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b79e875-5537-492e-aa94-3a015701a0f8-istiod-ca-cert\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.794652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.794528    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-certs\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.794746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.794650    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-credential-socket\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.796789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.796763    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-envoy\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.796912 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.796839    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b79e875-5537-492e-aa94-3a015701a0f8-istio-podinfo\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.818749 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.818727    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-istio-token\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.819037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.819013    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkq47\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-kube-api-access-fkq47\") pod \"e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") " pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:20.925364 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.922213    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:20.995210 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:20.994661    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:21.436146 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.436105    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"]
Oct 30 06:17:21.436731 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:17:21.436699    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b79e875_5537_492e_aa94_3a015701a0f8.slice/crio-682d00cbe2f9d5b7807772f869732d9aac00052485e5e44fff24f777779ba569 WatchSource:0}: Error finding container 682d00cbe2f9d5b7807772f869732d9aac00052485e5e44fff24f777779ba569: Status 404 returned error can't find the container with id 682d00cbe2f9d5b7807772f869732d9aac00052485e5e44fff24f777779ba569
Oct 30 06:17:21.440129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.440096    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:17:21.440231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.440165    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:17:21.440231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.440222    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:17:21.615103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.615065    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:21.650395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.650335    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" event={"ID":"3b79e875-5537-492e-aa94-3a015701a0f8","Type":"ContainerStarted","Data":"304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62"}
Oct 30 06:17:21.650597 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.650451    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" event={"ID":"3b79e875-5537-492e-aa94-3a015701a0f8","Type":"ContainerStarted","Data":"682d00cbe2f9d5b7807772f869732d9aac00052485e5e44fff24f777779ba569"}
Oct 30 06:17:21.705858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.705750    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podStartSLOduration=1.70571598 podStartE2EDuration="1.70571598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:21.704003689 +0000 UTC m=+2756.535040899" watchObservedRunningTime="2026-10-30 06:17:21.70571598 +0000 UTC m=+2756.536753197"
Oct 30 06:17:21.792206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:21.784074    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:22.008827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:22.005012    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:22.013337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:22.013302    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 06:17:22.013592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:22.013566    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 06:17:22.587463 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:22.587425    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:23.008066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:23.005979    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:23.334539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:23.334443    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:23.567113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:23.567083    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:23.668580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:23.668543    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:23.669687 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:23.669665    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:17:23.727977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:23.727944    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:24.100501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:24.100411    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:24.846982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:24.846954    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:25.000825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:25.000793    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:25.698240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:25.698206    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:26.759899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.759862    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:26.814093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.814058    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"]
Oct 30 06:17:26.818920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.818891    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:26.840549 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.840514    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"]
Oct 30 06:17:26.841611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.841581    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7fdb95\""
Oct 30 06:17:26.869645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.869612    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"]
Oct 30 06:17:26.874971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.874947    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:26.889548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.889520    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7fdb95-dockercfg-4gpnw\""
Oct 30 06:17:26.894544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.894515    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"]
Oct 30 06:17:26.917567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.917531    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:26.984204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.981951    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqkm\" (UniqueName: \"kubernetes.io/projected/07a44d46-bade-4663-98c1-d1e49bb857c7-kube-api-access-jpqkm\") pod \"payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz\" (UID: \"07a44d46-bade-4663-98c1-d1e49bb857c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:26.984204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:26.982262    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a44d46-bade-4663-98c1-d1e49bb857c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz\" (UID: \"07a44d46-bade-4663-98c1-d1e49bb857c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:27.108347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.105621    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k7vb\" (UniqueName: \"kubernetes.io/projected/108dde7f-dc08-4fe4-8ae8-dad940d32666-kube-api-access-9k7vb\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg\" (UID: \"108dde7f-dc08-4fe4-8ae8-dad940d32666\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:27.108347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.105725    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqkm\" (UniqueName: \"kubernetes.io/projected/07a44d46-bade-4663-98c1-d1e49bb857c7-kube-api-access-jpqkm\") pod \"payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz\" (UID: \"07a44d46-bade-4663-98c1-d1e49bb857c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:27.108347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.105895    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a44d46-bade-4663-98c1-d1e49bb857c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz\" (UID: \"07a44d46-bade-4663-98c1-d1e49bb857c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:27.108347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.105964    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108dde7f-dc08-4fe4-8ae8-dad940d32666-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg\" (UID: \"108dde7f-dc08-4fe4-8ae8-dad940d32666\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:27.117205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.117014    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a44d46-bade-4663-98c1-d1e49bb857c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz\" (UID: \"07a44d46-bade-4663-98c1-d1e49bb857c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:27.185245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.179421    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqkm\" (UniqueName: \"kubernetes.io/projected/07a44d46-bade-4663-98c1-d1e49bb857c7-kube-api-access-jpqkm\") pod \"payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz\" (UID: \"07a44d46-bade-4663-98c1-d1e49bb857c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:27.207302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.207262    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k7vb\" (UniqueName: \"kubernetes.io/projected/108dde7f-dc08-4fe4-8ae8-dad940d32666-kube-api-access-9k7vb\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg\" (UID: \"108dde7f-dc08-4fe4-8ae8-dad940d32666\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:27.207624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.207601    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108dde7f-dc08-4fe4-8ae8-dad940d32666-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg\" (UID: \"108dde7f-dc08-4fe4-8ae8-dad940d32666\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:27.208407 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.208383    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108dde7f-dc08-4fe4-8ae8-dad940d32666-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg\" (UID: \"108dde7f-dc08-4fe4-8ae8-dad940d32666\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:27.231082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.231053    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k7vb\" (UniqueName: \"kubernetes.io/projected/108dde7f-dc08-4fe4-8ae8-dad940d32666-kube-api-access-9k7vb\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg\" (UID: \"108dde7f-dc08-4fe4-8ae8-dad940d32666\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:27.438005 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.437971    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:27.491974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.491942    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:27.753501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.753471    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"]
Oct 30 06:17:27.753650 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:17:27.753633    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a44d46_bade_4663_98c1_d1e49bb857c7.slice/crio-6367b6cfa50e9c1f126fc96d8124121575bdc4364c8091786934498642438fb6 WatchSource:0}: Error finding container 6367b6cfa50e9c1f126fc96d8124121575bdc4364c8091786934498642438fb6: Status 404 returned error can't find the container with id 6367b6cfa50e9c1f126fc96d8124121575bdc4364c8091786934498642438fb6
Oct 30 06:17:27.835026 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.834997    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"]
Oct 30 06:17:27.837063 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:17:27.837037    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108dde7f_dc08_4fe4_8ae8_dad940d32666.slice/crio-68cebbb0bd799f379541ffe3fb559655182419821218e4af54da438d0cb15d21 WatchSource:0}: Error finding container 68cebbb0bd799f379541ffe3fb559655182419821218e4af54da438d0cb15d21: Status 404 returned error can't find the container with id 68cebbb0bd799f379541ffe3fb559655182419821218e4af54da438d0cb15d21
Oct 30 06:17:27.928101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:27.928078    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:28.710198 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:28.710148    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz" event={"ID":"07a44d46-bade-4663-98c1-d1e49bb857c7","Type":"ContainerStarted","Data":"922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a"}
Oct 30 06:17:28.710398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:28.710226    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz" event={"ID":"07a44d46-bade-4663-98c1-d1e49bb857c7","Type":"ContainerStarted","Data":"6367b6cfa50e9c1f126fc96d8124121575bdc4364c8091786934498642438fb6"}
Oct 30 06:17:28.711917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:28.711887    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg" event={"ID":"108dde7f-dc08-4fe4-8ae8-dad940d32666","Type":"ContainerStarted","Data":"b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f"}
Oct 30 06:17:28.712054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:28.711937    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg" event={"ID":"108dde7f-dc08-4fe4-8ae8-dad940d32666","Type":"ContainerStarted","Data":"68cebbb0bd799f379541ffe3fb559655182419821218e4af54da438d0cb15d21"}
Oct 30 06:17:28.712054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:28.711974    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:28.749039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:28.748982    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz" podStartSLOduration=2.748965707 podStartE2EDuration="2.748965707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:28.746722448 +0000 UTC m=+2763.577759669" watchObservedRunningTime="2026-10-30 06:17:28.748965707 +0000 UTC m=+2763.580002928"
Oct 30 06:17:28.786133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:28.786076    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg" podStartSLOduration=2.786058294 podStartE2EDuration="2.786058294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:28.7827886 +0000 UTC m=+2763.613825818" watchObservedRunningTime="2026-10-30 06:17:28.786058294 +0000 UTC m=+2763.617095514"
Oct 30 06:17:29.720817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:29.720784    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:29.788819 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:29.788786    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:30.197434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:30.197402    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:30.351757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:30.351660    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:30.460595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:30.460513    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"]
Oct 30 06:17:30.460783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:30.460761    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy" containerID="cri-o://de5ad4386b16dae4df5a92d4853ef57fe17998d4eee491c386e4991ecbfd6283" gracePeriod=30
Oct 30 06:17:30.545101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:30.545069    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:30.658206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:30.650624    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"]
Oct 30 06:17:30.794338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:30.791091    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:30.961694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:30.961656    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:31.917519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:31.917483    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:32.074050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:32.074020    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:32.374808 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:17:32.374729    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:17:33.836732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:33.836695    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:34.614918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:34.614874    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n" podUID="b72959fa-4c54-4917-9f9d-27d00c97d4ce" containerName="main" containerID="cri-o://998a84dd7101a59e9448de1312a2ba77b00e2f843baead3381d4f92ae7f4530b" gracePeriod=45
Oct 30 06:17:34.832140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:34.832105    5250 generic.go:356] "Generic (PLEG): container finished" podID="b72959fa-4c54-4917-9f9d-27d00c97d4ce" containerID="998a84dd7101a59e9448de1312a2ba77b00e2f843baead3381d4f92ae7f4530b" exitCode=0
Oct 30 06:17:34.832324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:34.832165    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n" event={"ID":"b72959fa-4c54-4917-9f9d-27d00c97d4ce","Type":"ContainerDied","Data":"998a84dd7101a59e9448de1312a2ba77b00e2f843baead3381d4f92ae7f4530b"}
Oct 30 06:17:34.895051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:34.895024    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:17:35.025646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.025617    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-home\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-home\") pod \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") "
Oct 30 06:17:35.025830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.025730    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-model-cache\") pod \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") "
Oct 30 06:17:35.025830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.025771    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tls-certs\") pod \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") "
Oct 30 06:17:35.025914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.025832    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tmp-dir\") pod \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") "
Oct 30 06:17:35.025914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.025870    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-home" pod "b72959fa-4c54-4917-9f9d-27d00c97d4ce" (UID: "b72959fa-4c54-4917-9f9d-27d00c97d4ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:35.025914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.025904    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-dshm\") pod \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") "
Oct 30 06:17:35.026063 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.026014    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-model-cache" pod "b72959fa-4c54-4917-9f9d-27d00c97d4ce" (UID: "b72959fa-4c54-4917-9f9d-27d00c97d4ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:35.026116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.026083    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tmp-dir" pod "b72959fa-4c54-4917-9f9d-27d00c97d4ce" (UID: "b72959fa-4c54-4917-9f9d-27d00c97d4ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:35.026151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.026116    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72959fa-4c54-4917-9f9d-27d00c97d4ce-kube-api-access-4wrql\" (UniqueName: \"kubernetes.io/projected/b72959fa-4c54-4917-9f9d-27d00c97d4ce-kube-api-access-4wrql\") pod \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\" (UID: \"b72959fa-4c54-4917-9f9d-27d00c97d4ce\") "
Oct 30 06:17:35.026781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.026759    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:35.026781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.026780    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:35.026989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.026792    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:35.028399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.028372    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-dshm" pod "b72959fa-4c54-4917-9f9d-27d00c97d4ce" (UID: "b72959fa-4c54-4917-9f9d-27d00c97d4ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:35.028495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.028475    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72959fa-4c54-4917-9f9d-27d00c97d4ce-kube-api-access-4wrql" pod "b72959fa-4c54-4917-9f9d-27d00c97d4ce" (UID: "b72959fa-4c54-4917-9f9d-27d00c97d4ce"). InnerVolumeSpecName "kube-api-access-4wrql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:17:35.028556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.028496    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tls-certs" pod "b72959fa-4c54-4917-9f9d-27d00c97d4ce" (UID: "b72959fa-4c54-4917-9f9d-27d00c97d4ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:17:35.128389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.128252    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72959fa-4c54-4917-9f9d-27d00c97d4ce-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:35.128389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.128301    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72959fa-4c54-4917-9f9d-27d00c97d4ce-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:35.128389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.128317    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wrql\" (UniqueName: \"kubernetes.io/projected/b72959fa-4c54-4917-9f9d-27d00c97d4ce-kube-api-access-4wrql\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:35.175808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.175776    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:35.368372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.367482    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:35.514978 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.514943    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:35.656387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.656268    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6" podUID="77f849b1-2e44-4485-bb6c-01c53cdaee28" containerName="payload-pre-processing" containerID="cri-o://4d44ca6687c4b857574250a6c519832f3a297a5985c7f7571a4635996d7d15d6" gracePeriod=25
Oct 30 06:17:35.711440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.711369    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:35.860237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.860147    5250 generic.go:356] "Generic (PLEG): container finished" podID="77f849b1-2e44-4485-bb6c-01c53cdaee28" containerID="4d44ca6687c4b857574250a6c519832f3a297a5985c7f7571a4635996d7d15d6" exitCode=0
Oct 30 06:17:35.860490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.860241    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6" event={"ID":"77f849b1-2e44-4485-bb6c-01c53cdaee28","Type":"ContainerDied","Data":"4d44ca6687c4b857574250a6c519832f3a297a5985c7f7571a4635996d7d15d6"}
Oct 30 06:17:35.862793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.862762    5250 generic.go:356] "Generic (PLEG): container finished" podID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerID="de5ad4386b16dae4df5a92d4853ef57fe17998d4eee491c386e4991ecbfd6283" exitCode=0
Oct 30 06:17:35.862933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.862832    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" event={"ID":"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b","Type":"ContainerDied","Data":"de5ad4386b16dae4df5a92d4853ef57fe17998d4eee491c386e4991ecbfd6283"}
Oct 30 06:17:35.865146 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.865125    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n" event={"ID":"b72959fa-4c54-4917-9f9d-27d00c97d4ce","Type":"ContainerDied","Data":"6404e9e9278b6433a26e2889d5755c3ad00d590ab2337cf14862b57d2c55b9e7"}
Oct 30 06:17:35.865288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.865156    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"
Oct 30 06:17:35.865288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.865160    5250 scope.go:122] "RemoveContainer" containerID="998a84dd7101a59e9448de1312a2ba77b00e2f843baead3381d4f92ae7f4530b"
Oct 30 06:17:35.918137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.918097    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"]
Oct 30 06:17:35.938355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:35.938326    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2f1e4c/sub-test-model-2f1e4c-kserve-959978675-grb7n"]
Oct 30 06:17:36.013060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.012990    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:36.060669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.060638    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:17:36.151344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.151305    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f849b1-2e44-4485-bb6c-01c53cdaee28-kube-api-access-6tw5v\" (UniqueName: \"kubernetes.io/projected/77f849b1-2e44-4485-bb6c-01c53cdaee28-kube-api-access-6tw5v\") pod \"77f849b1-2e44-4485-bb6c-01c53cdaee28\" (UID: \"77f849b1-2e44-4485-bb6c-01c53cdaee28\") "
Oct 30 06:17:36.151539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.151403    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f849b1-2e44-4485-bb6c-01c53cdaee28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f849b1-2e44-4485-bb6c-01c53cdaee28-plugins-config-volume\") pod \"77f849b1-2e44-4485-bb6c-01c53cdaee28\" (UID: \"77f849b1-2e44-4485-bb6c-01c53cdaee28\") "
Oct 30 06:17:36.153722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.153687    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f849b1-2e44-4485-bb6c-01c53cdaee28-plugins-config-volume" pod "77f849b1-2e44-4485-bb6c-01c53cdaee28" (UID: "77f849b1-2e44-4485-bb6c-01c53cdaee28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:17:36.156755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.156726    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f849b1-2e44-4485-bb6c-01c53cdaee28-kube-api-access-6tw5v" pod "77f849b1-2e44-4485-bb6c-01c53cdaee28" (UID: "77f849b1-2e44-4485-bb6c-01c53cdaee28"). InnerVolumeSpecName "kube-api-access-6tw5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:17:36.160465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.160440    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:36.182442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.182416    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 06:17:36.254988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.254953    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tw5v\" (UniqueName: \"kubernetes.io/projected/77f849b1-2e44-4485-bb6c-01c53cdaee28-kube-api-access-6tw5v\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.255152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.254993    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f849b1-2e44-4485-bb6c-01c53cdaee28-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.355735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.355644    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-token\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-token\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.355735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.355688    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istiod-ca-cert\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.355974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.355736    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-credential-socket\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.355974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.355819    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-kube-api-access-j294c\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-kube-api-access-j294c\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.355974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.355892    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-data\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.356142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356003    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-podinfo\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.356142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356044    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-socket\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.356142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356107    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-credential-socket" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:36.356415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356200    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-data" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:36.356415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356248    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istiod-ca-cert" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:17:36.356415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356284    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-envoy\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.356415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356332    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-certs\") pod \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\" (UID: \"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b\") "
Oct 30 06:17:36.356619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356436    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-socket" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:36.356712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.356688    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-certs" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:36.357054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.357001    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.357054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.357022    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.357054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.357031    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.357054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.357039    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.357054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.357048    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.358470 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.358435    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-podinfo" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:17:36.358580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.358521    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-envoy" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:36.358740 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.358716    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-kube-api-access-j294c" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "kube-api-access-j294c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:17:36.358879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.358749    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-token" pod "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" (UID: "f9d40622-3e44-4a2c-9029-d12f3ebe2d6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:17:36.458383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.458255    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.458383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.458289    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.458383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.458303    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j294c\" (UniqueName: \"kubernetes.io/projected/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-kube-api-access-j294c\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.458383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.458321    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:36.541053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.541022    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:36.693412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.693378    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:36.873790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.873762    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"
Oct 30 06:17:36.873944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.873810    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l" event={"ID":"f9d40622-3e44-4a2c-9029-d12f3ebe2d6b","Type":"ContainerDied","Data":"6d04c7226ec8bf1dfdb1ee78ed98d89487c42e11606bb228e74162fa2feb6add"}
Oct 30 06:17:36.873944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.873868    5250 scope.go:122] "RemoveContainer" containerID="de5ad4386b16dae4df5a92d4853ef57fe17998d4eee491c386e4991ecbfd6283"
Oct 30 06:17:36.876754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.876727    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6" event={"ID":"77f849b1-2e44-4485-bb6c-01c53cdaee28","Type":"ContainerDied","Data":"8666f803be6b82185846d2e3e92e9e231f6cbbe1b579e48785d229655cecc60c"}
Oct 30 06:17:36.876882 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.876764    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"
Oct 30 06:17:36.887154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.887123    5250 scope.go:122] "RemoveContainer" containerID="4d44ca6687c4b857574250a6c519832f3a297a5985c7f7571a4635996d7d15d6"
Oct 30 06:17:36.974793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.974752    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"]
Oct 30 06:17:36.979984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:36.979957    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-5445db4799-vznw6"]
Oct 30 06:17:37.006323 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:37.006296    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"]
Oct 30 06:17:37.010378 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:37.010352    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-rmv9l"]
Oct 30 06:17:37.283457 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:37.283378    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f849b1-2e44-4485-bb6c-01c53cdaee28" path="/var/lib/kubelet/pods/77f849b1-2e44-4485-bb6c-01c53cdaee28/volumes"
Oct 30 06:17:37.283839 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:37.283825    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72959fa-4c54-4917-9f9d-27d00c97d4ce" path="/var/lib/kubelet/pods/b72959fa-4c54-4917-9f9d-27d00c97d4ce/volumes"
Oct 30 06:17:37.284287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:37.284274    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" path="/var/lib/kubelet/pods/f9d40622-3e44-4a2c-9029-d12f3ebe2d6b/volumes"
Oct 30 06:17:37.455395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:37.455359    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:37.613373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:37.613296    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:38.554989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:38.554904    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:39.724371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:39.724340    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:17:40.505703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:40.505671    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:40.791892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:40.791812    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:17:42.300845 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:42.300809    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:42.638159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:42.638125    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:42.780491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:42.780460    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:42.984514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:42.984408    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:43.977878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:43.977839    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"]
Oct 30 06:17:43.978405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:43.978060    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy" containerID="cri-o://8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf" gracePeriod=30
Oct 30 06:17:44.567869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.567836    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:44.662382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.662346    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"]
Oct 30 06:17:44.665384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665283    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy"
Oct 30 06:17:44.665384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665312    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy"
Oct 30 06:17:44.665384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665373    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b72959fa-4c54-4917-9f9d-27d00c97d4ce" containerName="main"
Oct 30 06:17:44.665384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665381    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72959fa-4c54-4917-9f9d-27d00c97d4ce" containerName="main"
Oct 30 06:17:44.665746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665431    5250 cpu_manager.go:414] "containerMap: removing container" podUID="77f849b1-2e44-4485-bb6c-01c53cdaee28" containerName="payload-pre-processing"
Oct 30 06:17:44.665746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665441    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f849b1-2e44-4485-bb6c-01c53cdaee28" containerName="payload-pre-processing"
Oct 30 06:17:44.665746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665621    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72959fa-4c54-4917-9f9d-27d00c97d4ce" containerName="main"
Oct 30 06:17:44.665746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665646    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f849b1-2e44-4485-bb6c-01c53cdaee28" containerName="payload-pre-processing"
Oct 30 06:17:44.665746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.665659    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d40622-3e44-4a2c-9029-d12f3ebe2d6b" containerName="istio-proxy"
Oct 30 06:17:44.672393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.672368    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.690089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.690061    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"]
Oct 30 06:17:44.691268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.691206    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-36ae27\"/\"openshift-service-ca.crt\""
Oct 30 06:17:44.691374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.691272    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-36ae27\"/\"kube-root-ca.crt\""
Oct 30 06:17:44.696358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.696327    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-36ae27\"/\"default-dockercfg-l45rb\""
Oct 30 06:17:44.696504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.696410    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-36ae27\"/\"ipp-route-36ae27-kserve-self-signed-certs\""
Oct 30 06:17:44.747864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.747797    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:44.796428 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.796386    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-model-cache\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.796624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.796522    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xn9\" (UniqueName: \"kubernetes.io/projected/982646c7-61c8-40f6-851f-c599a18f7920-kube-api-access-f2xn9\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.796690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.796625    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-dshm\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.796690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.796669    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982646c7-61c8-40f6-851f-c599a18f7920-tls-certs\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.796819 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.796728    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-home\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.796819 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.796793    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-tmp-dir\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.898470 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.898433    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xn9\" (UniqueName: \"kubernetes.io/projected/982646c7-61c8-40f6-851f-c599a18f7920-kube-api-access-f2xn9\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.898681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.898501    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-dshm\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.898681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.898547    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982646c7-61c8-40f6-851f-c599a18f7920-tls-certs\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.898681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.898592    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-home\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.898681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.898658    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-tmp-dir\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.898927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.898822    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-model-cache\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.899132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.899105    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-home\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.899410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.899161    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-model-cache\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.899410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.899269    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-tmp-dir\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.901212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.901172    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-dshm\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.902074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.902007    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982646c7-61c8-40f6-851f-c599a18f7920-tls-certs\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:44.954207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:44.953456    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xn9\" (UniqueName: \"kubernetes.io/projected/982646c7-61c8-40f6-851f-c599a18f7920-kube-api-access-f2xn9\") pod \"ipp-route-36ae27-kserve-7f74b87ff7-tzdgw\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") " pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:45.018205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:45.017148    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:45.400653 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:45.400623    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"]
Oct 30 06:17:45.401335 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:17:45.401303    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod982646c7_61c8_40f6_851f_c599a18f7920.slice/crio-f3bf1a97f0fe5edd836c33dfe01452580ccabc10ceb04652b36ed2efbe335f06 WatchSource:0}: Error finding container f3bf1a97f0fe5edd836c33dfe01452580ccabc10ceb04652b36ed2efbe335f06: Status 404 returned error can't find the container with id f3bf1a97f0fe5edd836c33dfe01452580ccabc10ceb04652b36ed2efbe335f06
Oct 30 06:17:46.035730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:46.035687    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw" event={"ID":"982646c7-61c8-40f6-851f-c599a18f7920","Type":"ContainerStarted","Data":"d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28"}
Oct 30 06:17:46.036147 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:46.035741    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw" event={"ID":"982646c7-61c8-40f6-851f-c599a18f7920","Type":"ContainerStarted","Data":"f3bf1a97f0fe5edd836c33dfe01452580ccabc10ceb04652b36ed2efbe335f06"}
Oct 30 06:17:46.077324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:46.077272    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw" podStartSLOduration=2.077258596 podStartE2EDuration="2.077258596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:46.073287906 +0000 UTC m=+2780.904325122" watchObservedRunningTime="2026-10-30 06:17:46.077258596 +0000 UTC m=+2780.908295813"
Oct 30 06:17:46.422397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:46.422365    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:47.018518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:47.018482    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:48.127343 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:48.127307    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:49.263778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.263753    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:17:49.391807 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.391778    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-socket\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.391860    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-data\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.391961    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-kube-api-access-5jczc\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-kube-api-access-5jczc\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392027    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-envoy\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392044    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-socket" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:49.392297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392100    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-data" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:49.392297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392110    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-credential-socket\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392152    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-certs\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392194    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istiod-ca-cert\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392279    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-podinfo\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392350    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-certs" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:49.392594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392406    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-credential-socket" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:49.392594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392474    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-token\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-token\") pod \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\" (UID: \"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb\") "
Oct 30 06:17:49.392782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.392709    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istiod-ca-cert" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:17:49.393701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.393671    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.393840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.393704    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.393840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.393719    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.393840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.393733    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.393840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.393747    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.394659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.394631    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-kube-api-access-5jczc" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "kube-api-access-5jczc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:17:49.394770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.394659    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-envoy" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:17:49.394770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.394742    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-podinfo" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:17:49.395045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.395028    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-token" pod "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" (UID: "21f9fb67-bc26-41f0-bd53-1e63f3f9acdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:17:49.495642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.495564    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jczc\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-kube-api-access-5jczc\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.495642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.495597    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.495642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.495608    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.495642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.495615    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:17:49.777536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:49.777450    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:50.070713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.070622    5250 generic.go:356] "Generic (PLEG): container finished" podID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerID="8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf" exitCode=0
Oct 30 06:17:50.070713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.070709    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"
Oct 30 06:17:50.070936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.070713    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" event={"ID":"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb","Type":"ContainerDied","Data":"8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf"}
Oct 30 06:17:50.070936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.070800    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg" event={"ID":"21f9fb67-bc26-41f0-bd53-1e63f3f9acdb","Type":"ContainerDied","Data":"c25c49476ca2dbfda98545c0f5c23d0f705999c10a79178811617be9e2c6099e"}
Oct 30 06:17:50.070936 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.070822    5250 scope.go:122] "RemoveContainer" containerID="8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf"
Oct 30 06:17:50.086484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.086464    5250 scope.go:122] "RemoveContainer" containerID="8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf"
Oct 30 06:17:50.086794 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:17:50.086772    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf\": container with ID starting with 8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf not found: ID does not exist" containerID="8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf"
Oct 30 06:17:50.086875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.086806    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf"} err="failed to get container status \"8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf\": rpc error: code = NotFound desc = could not find container \"8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf\": container with ID starting with 8f0b4067681b689d53eedd821182167c9f1a4e5f5262ca4dcaa2ab7a5c4ea2bf not found: ID does not exist"
Oct 30 06:17:50.119906 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.119870    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"]
Oct 30 06:17:50.130575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:50.130546    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-dwnsg"]
Oct 30 06:17:51.283931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:51.283899    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" path="/var/lib/kubelet/pods/21f9fb67-bc26-41f0-bd53-1e63f3f9acdb/volumes"
Oct 30 06:17:51.588521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:51.588433    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:53.190389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:53.190361    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:55.017831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:55.017795    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:55.030551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:55.030528    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:55.125267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:55.125235    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:17:55.213630 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:55.213595    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:56.670585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:56.670549    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:56.840400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:56.840364    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:56.996992 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:56.996886    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:57.310297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:57.305797    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:57.455065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:57.455035    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:57.596646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:57.596548    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:57.953887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:57.953857    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:17:59.158507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:17:59.158470    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:01.354532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:01.354489    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:02.375955 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:18:02.375922    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:18:03.379477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:03.379441    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:04.125871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:04.125835    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:04.272161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:04.272126    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:04.459342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:04.459307    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:05.121628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:05.121574    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:06.941290 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:06.941167    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:08.576663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:08.576624    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:10.219348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:10.219304    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:12.463753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:12.463714    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:14.083619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:14.083582    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:14.265771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:14.265732    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:15.522582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:15.522519    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:15.674211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:15.674163    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:15.825426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:15.825312    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:16.710397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:16.710364    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:18.398872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:18.398829    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:19.937820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:19.937789    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:21.195967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:21.195932    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:21.343230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:21.343167    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:21.655358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:21.655321    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:23.256217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:23.256146    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:24.889725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:24.889694    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"]
Oct 30 06:18:25.039347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:25.039315    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:25.186004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:25.185925    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:25.328926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:25.328890    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:27.129087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:27.129051    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:28.906919 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:28.906878    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:28.944948 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:28.944910    5250 scope.go:122] "RemoveContainer" containerID="6287c720bc265b45af39f0bdf4683303173dd9d694c998179b94872e930d359c"
Oct 30 06:18:29.891536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:29.891447    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns" podUID="e15b7328-9cdf-413c-ba5e-cd1f6821687a" containerName="payload-processing" containerID="cri-o://f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014" gracePeriod=25
Oct 30 06:18:30.189427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.189399    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:18:30.328306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.328276    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15b7328-9cdf-413c-ba5e-cd1f6821687a-kube-api-access-7dpkh\" (UniqueName: \"kubernetes.io/projected/e15b7328-9cdf-413c-ba5e-cd1f6821687a-kube-api-access-7dpkh\") pod \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\" (UID: \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\") "
Oct 30 06:18:30.328486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.328357    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15b7328-9cdf-413c-ba5e-cd1f6821687a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15b7328-9cdf-413c-ba5e-cd1f6821687a-plugins-config-volume\") pod \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\" (UID: \"e15b7328-9cdf-413c-ba5e-cd1f6821687a\") "
Oct 30 06:18:30.328718 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.328693    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15b7328-9cdf-413c-ba5e-cd1f6821687a-plugins-config-volume" pod "e15b7328-9cdf-413c-ba5e-cd1f6821687a" (UID: "e15b7328-9cdf-413c-ba5e-cd1f6821687a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:18:30.329131 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.329063    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15b7328-9cdf-413c-ba5e-cd1f6821687a-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:30.330686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.330661    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15b7328-9cdf-413c-ba5e-cd1f6821687a-kube-api-access-7dpkh" pod "e15b7328-9cdf-413c-ba5e-cd1f6821687a" (UID: "e15b7328-9cdf-413c-ba5e-cd1f6821687a"). InnerVolumeSpecName "kube-api-access-7dpkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:18:30.430138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.430064    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dpkh\" (UniqueName: \"kubernetes.io/projected/e15b7328-9cdf-413c-ba5e-cd1f6821687a-kube-api-access-7dpkh\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:30.593202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.593144    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:30.642614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.642574    5250 generic.go:356] "Generic (PLEG): container finished" podID="e15b7328-9cdf-413c-ba5e-cd1f6821687a" containerID="f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014" exitCode=0
Oct 30 06:18:30.642879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.642652    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"
Oct 30 06:18:30.642879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.642671    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns" event={"ID":"e15b7328-9cdf-413c-ba5e-cd1f6821687a","Type":"ContainerDied","Data":"f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014"}
Oct 30 06:18:30.642879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.642754    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns" event={"ID":"e15b7328-9cdf-413c-ba5e-cd1f6821687a","Type":"ContainerDied","Data":"dd4a87920d89b1a394c911afcf65a25be45643582b98da3b67e6a74a937005da"}
Oct 30 06:18:30.642879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.642802    5250 scope.go:122] "RemoveContainer" containerID="f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014"
Oct 30 06:18:30.656833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.656812    5250 scope.go:122] "RemoveContainer" containerID="f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014"
Oct 30 06:18:30.657134 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:18:30.657117    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014\": container with ID starting with f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014 not found: ID does not exist" containerID="f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014"
Oct 30 06:18:30.657190 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.657145    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014"} err="failed to get container status \"f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014\": rpc error: code = NotFound desc = could not find container \"f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014\": container with ID starting with f9f667abd624acb73296ab159bda2c83d8b665ebdf017139448b297e8aaea014 not found: ID does not exist"
Oct 30 06:18:30.691937 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.691866    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"]
Oct 30 06:18:30.696269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:30.696239    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-8cdf67d76-llrns"]
Oct 30 06:18:31.283333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:31.283300    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15b7328-9cdf-413c-ba5e-cd1f6821687a" path="/var/lib/kubelet/pods/e15b7328-9cdf-413c-ba5e-cd1f6821687a/volumes"
Oct 30 06:18:32.374602 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:18:32.374569    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:18:32.429839 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:32.429807    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:33.604835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:33.604800    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:34.078709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:34.078673    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:34.532814 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:34.532777    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:35.612861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:35.612823    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:36.385769 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:36.385731    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:36.604024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:36.603986    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:36.747327 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:36.747202    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:36.903830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:36.903794    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:37.173149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:37.173100    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:37.326364 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:37.326328    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:37.982367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:37.982334    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:38.140216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:38.140161    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:38.987905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:38.987873    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:39.580877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:39.580773    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:39.727776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:39.727739    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:40.897592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:40.897554    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:42.968094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:42.968064    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:43.222345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:43.222264    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:43.425666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:43.425627    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:43.572254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:43.572138    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:44.276263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.276228    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"]
Oct 30 06:18:44.277014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.276993    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e15b7328-9cdf-413c-ba5e-cd1f6821687a" containerName="payload-processing"
Oct 30 06:18:44.277014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.277014    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15b7328-9cdf-413c-ba5e-cd1f6821687a" containerName="payload-processing"
Oct 30 06:18:44.277126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.277025    5250 cpu_manager.go:414] "containerMap: removing container" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy"
Oct 30 06:18:44.277126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.277030    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy"
Oct 30 06:18:44.277277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.277157    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15b7328-9cdf-413c-ba5e-cd1f6821687a" containerName="payload-processing"
Oct 30 06:18:44.277277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.277165    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f9fb67-bc26-41f0-bd53-1e63f3f9acdb" containerName="istio-proxy"
Oct 30 06:18:44.280895 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.280878    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.301894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.301861    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"]
Oct 30 06:18:44.303455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.303392    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:18:44.303725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.303707    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:18:44.364398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.364364    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.364571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.364542    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.364710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.364694    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4x4b\" (UniqueName: \"kubernetes.io/projected/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-kube-api-access-w4x4b\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.466280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.466250    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.466484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.466372    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4x4b\" (UniqueName: \"kubernetes.io/projected/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-kube-api-access-w4x4b\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.466484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.466435    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.467150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.467124    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.469093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.469068    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.487681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.487649    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4x4b\" (UniqueName: \"kubernetes.io/projected/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-kube-api-access-w4x4b\") pod \"maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.597236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.597150    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:44.870521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.870424    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:44.917459 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:18:44.917423    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d77e321_e8ce_49e5_98e0_34f236d8cfa1.slice/crio-aeb4a05a39c4409ce6b6833b220d159196a13b656f2aa44b131ed50f7b551664 WatchSource:0}: Error finding container aeb4a05a39c4409ce6b6833b220d159196a13b656f2aa44b131ed50f7b551664: Status 404 returned error can't find the container with id aeb4a05a39c4409ce6b6833b220d159196a13b656f2aa44b131ed50f7b551664
Oct 30 06:18:44.922273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:44.922243    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"]
Oct 30 06:18:45.790334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:45.790300    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4" event={"ID":"4d77e321-e8ce-49e5-98e0-34f236d8cfa1","Type":"ContainerStarted","Data":"12ff6f703d1aad834c468260bce13bfef0490d3149f8b4ec722962731ba6fb89"}
Oct 30 06:18:45.790744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:45.790353    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4" event={"ID":"4d77e321-e8ce-49e5-98e0-34f236d8cfa1","Type":"ContainerStarted","Data":"aeb4a05a39c4409ce6b6833b220d159196a13b656f2aa44b131ed50f7b551664"}
Oct 30 06:18:45.829300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:45.829248    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4" podStartSLOduration=1.8292354199999998 podStartE2EDuration="1.82923542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:45.826267061 +0000 UTC m=+2840.657304277" watchObservedRunningTime="2026-10-30 06:18:45.82923542 +0000 UTC m=+2840.660272637"
Oct 30 06:18:46.566389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.566355    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:46.800022 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.799987    5250 generic.go:356] "Generic (PLEG): container finished" podID="3b79e875-5537-492e-aa94-3a015701a0f8" containerID="304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62" exitCode=0
Oct 30 06:18:46.800535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.800063    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" event={"ID":"3b79e875-5537-492e-aa94-3a015701a0f8","Type":"ContainerDied","Data":"304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62"}
Oct 30 06:18:46.800644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.800617    5250 scope.go:122] "RemoveContainer" containerID="304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62"
Oct 30 06:18:46.801860 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.801838    5250 generic.go:356] "Generic (PLEG): container finished" podID="4d77e321-e8ce-49e5-98e0-34f236d8cfa1" containerID="12ff6f703d1aad834c468260bce13bfef0490d3149f8b4ec722962731ba6fb89" exitCode=0
Oct 30 06:18:46.801960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.801874    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4" event={"ID":"4d77e321-e8ce-49e5-98e0-34f236d8cfa1","Type":"ContainerDied","Data":"12ff6f703d1aad834c468260bce13bfef0490d3149f8b4ec722962731ba6fb89"}
Oct 30 06:18:46.802629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.802601    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:18:46.802688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.802673    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:18:46.802734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:46.802704    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:18:47.032782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.032748    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"]
Oct 30 06:18:47.037585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.037556    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661c28d-vng59"]
Oct 30 06:18:47.289156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.289100    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71cccff8-cf87-497e-aeee-ca0bf0ff44d3" path="/var/lib/kubelet/pods/71cccff8-cf87-497e-aeee-ca0bf0ff44d3/volumes"
Oct 30 06:18:47.811562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.811527    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" event={"ID":"3b79e875-5537-492e-aa94-3a015701a0f8","Type":"ContainerStarted","Data":"63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7"}
Oct 30 06:18:47.858864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.849205    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"]
Oct 30 06:18:47.858864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.849408    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm" podUID="6ea4e14a-9d8b-4460-8bfb-21f904305667" containerName="maas-api" containerID="cri-o://f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e" gracePeriod=30
Oct 30 06:18:47.945546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.938237    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:47.999118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.998122    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 06:18:47.999118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.998225    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 06:18:47.999118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:47.998523    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:18:48.120792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.120682    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:48.205275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.205241    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:48.297261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.290902    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:48.297261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.292672    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-xks-ca\") pod \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") "
Oct 30 06:18:48.297261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.292748    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-kube-api-access-w4x4b\" (UniqueName: \"kubernetes.io/projected/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-kube-api-access-w4x4b\") pod \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") "
Oct 30 06:18:48.297261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.292985    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-openshift-ca\") pod \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\" (UID: \"4d77e321-e8ce-49e5-98e0-34f236d8cfa1\") "
Oct 30 06:18:48.297261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.293950    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-openshift-ca" pod "4d77e321-e8ce-49e5-98e0-34f236d8cfa1" (UID: "4d77e321-e8ce-49e5-98e0-34f236d8cfa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:18:48.463234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.461406    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-kube-api-access-w4x4b" pod "4d77e321-e8ce-49e5-98e0-34f236d8cfa1" (UID: "4d77e321-e8ce-49e5-98e0-34f236d8cfa1"). InnerVolumeSpecName "kube-api-access-w4x4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:18:48.486203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.471603    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-xks-ca" pod "4d77e321-e8ce-49e5-98e0-34f236d8cfa1" (UID: "4d77e321-e8ce-49e5-98e0-34f236d8cfa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:18:48.486203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.480752    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:48.486203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.480786    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4x4b\" (UniqueName: \"kubernetes.io/projected/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-kube-api-access-w4x4b\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:48.486203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.480809    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d77e321-e8ce-49e5-98e0-34f236d8cfa1-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:48.523032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.521722    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:48.635220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.635162    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:18:48.696071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.695996    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:48.803202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.802624    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-metrics-certs\") pod \"6ea4e14a-9d8b-4460-8bfb-21f904305667\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") "
Oct 30 06:18:48.804280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.804251    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea4e14a-9d8b-4460-8bfb-21f904305667-kube-api-access-h9vj7\" (UniqueName: \"kubernetes.io/projected/6ea4e14a-9d8b-4460-8bfb-21f904305667-kube-api-access-h9vj7\") pod \"6ea4e14a-9d8b-4460-8bfb-21f904305667\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") "
Oct 30 06:18:48.804442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.804296    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea4e14a-9d8b-4460-8bfb-21f904305667-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea4e14a-9d8b-4460-8bfb-21f904305667-openshift-service-ca\") pod \"6ea4e14a-9d8b-4460-8bfb-21f904305667\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") "
Oct 30 06:18:48.804442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.804359    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls\") pod \"6ea4e14a-9d8b-4460-8bfb-21f904305667\" (UID: \"6ea4e14a-9d8b-4460-8bfb-21f904305667\") "
Oct 30 06:18:48.805588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.805553    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea4e14a-9d8b-4460-8bfb-21f904305667-openshift-service-ca" pod "6ea4e14a-9d8b-4460-8bfb-21f904305667" (UID: "6ea4e14a-9d8b-4460-8bfb-21f904305667"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:18:48.808542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.808394    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-metrics-certs" pod "6ea4e14a-9d8b-4460-8bfb-21f904305667" (UID: "6ea4e14a-9d8b-4460-8bfb-21f904305667"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:18:48.808542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.808426    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea4e14a-9d8b-4460-8bfb-21f904305667-kube-api-access-h9vj7" pod "6ea4e14a-9d8b-4460-8bfb-21f904305667" (UID: "6ea4e14a-9d8b-4460-8bfb-21f904305667"). InnerVolumeSpecName "kube-api-access-h9vj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:18:48.811513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.810399    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls" pod "6ea4e14a-9d8b-4460-8bfb-21f904305667" (UID: "6ea4e14a-9d8b-4460-8bfb-21f904305667"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:18:48.826927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.826891    5250 generic.go:356] "Generic (PLEG): container finished" podID="6ea4e14a-9d8b-4460-8bfb-21f904305667" containerID="f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e" exitCode=0
Oct 30 06:18:48.827404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.827024    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm" event={"ID":"6ea4e14a-9d8b-4460-8bfb-21f904305667","Type":"ContainerDied","Data":"f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e"}
Oct 30 06:18:48.827404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.827092    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm" event={"ID":"6ea4e14a-9d8b-4460-8bfb-21f904305667","Type":"ContainerDied","Data":"6e197691a5b8d53cd1e65273b2ffe25638a541fea57bbe461e7644f5b5510dfc"}
Oct 30 06:18:48.827404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.827113    5250 scope.go:122] "RemoveContainer" containerID="f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e"
Oct 30 06:18:48.827404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.827284    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"
Oct 30 06:18:48.833844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.833817    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"
Oct 30 06:18:48.836436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.836309    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4" event={"ID":"4d77e321-e8ce-49e5-98e0-34f236d8cfa1","Type":"ContainerDied","Data":"aeb4a05a39c4409ce6b6833b220d159196a13b656f2aa44b131ed50f7b551664"}
Oct 30 06:18:48.836436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.836378    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeb4a05a39c4409ce6b6833b220d159196a13b656f2aa44b131ed50f7b551664"
Oct 30 06:18:48.838070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.838026    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:48.844088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.844055    5250 scope.go:122] "RemoveContainer" containerID="f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e"
Oct 30 06:18:48.844499 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:18:48.844478    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e\": container with ID starting with f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e not found: ID does not exist" containerID="f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e"
Oct 30 06:18:48.844604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.844506    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e"} err="failed to get container status \"f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e\": rpc error: code = NotFound desc = could not find container \"f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e\": container with ID starting with f930a456dc226aa1858bca80f1c40b400324b554404340f639850c904fa32b4e not found: ID does not exist"
Oct 30 06:18:48.894252 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.894218    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"]
Oct 30 06:18:48.896100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.896073    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebe61e-7c7d98f5cf-wmpwm"]
Oct 30 06:18:48.906559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.906536    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:48.906559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.906560    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9vj7\" (UniqueName: \"kubernetes.io/projected/6ea4e14a-9d8b-4460-8bfb-21f904305667-kube-api-access-h9vj7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:48.906718 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.906571    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea4e14a-9d8b-4460-8bfb-21f904305667-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:48.906718 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.906581    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea4e14a-9d8b-4460-8bfb-21f904305667-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:48.996538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.996447    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 06:18:48.996538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:48.996509    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 06:18:49.220081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:49.220047    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:49.288082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:49.288001    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea4e14a-9d8b-4460-8bfb-21f904305667" path="/var/lib/kubelet/pods/6ea4e14a-9d8b-4460-8bfb-21f904305667/volumes"
Oct 30 06:18:49.995929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:49.995895    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 06:18:49.996387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:49.995948    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 06:18:50.509631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:50.509597    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:50.995587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:50.995555    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 06:18:50.995748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:50.995605    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 06:18:51.995909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:51.995869    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 06:18:51.996398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:51.995930    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 06:18:52.115313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:52.115278    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:52.995580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:52.995545    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 06:18:52.995751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:52.995600    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 06:18:53.668983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:53.668950    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:18:53.769528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:53.769494    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:54.001262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:54.001148    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:18:54.002320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:54.002298    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:18:54.414415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:54.414377    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"]
Oct 30 06:18:54.441899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:54.441868    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"]
Oct 30 06:18:54.573172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:54.573139    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:54.721053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:54.720965    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:55.570164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:55.570129    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:55.753415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:55.753376    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:55.896719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:55.896682    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:56.020661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:56.020625    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"]
Oct 30 06:18:56.020913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:56.020884    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy" containerID="cri-o://049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e" gracePeriod=30
Oct 30 06:18:56.091688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:56.091659    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:57.040889 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:57.040853    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"]
Oct 30 06:18:57.048414 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:57.048387    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07b940a-vwng7"]
Oct 30 06:18:57.283380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:57.283347    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf0a329-84b3-476f-9ca4-d064097e9814" path="/var/lib/kubelet/pods/1cf0a329-84b3-476f-9ca4-d064097e9814/volumes"
Oct 30 06:18:57.602849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:57.602807    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:18:57.603032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:57.602889    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:18:57.781462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:57.781429    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:59.415421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.415366    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg" podUID="c2dd459d-f6af-45f3-820d-f57ce3409e29" containerName="payload-processing" containerID="cri-o://fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c" gracePeriod=25
Oct 30 06:18:59.443024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.442990    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx" podUID="4fda6e76-0dd9-4775-99ad-9242db7d63cd" containerName="payload-pre-processing" containerID="cri-o://a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a" gracePeriod=25
Oct 30 06:18:59.507292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.507255    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:18:59.736519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.736485    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 06:18:59.785900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.785859    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 06:18:59.838527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.838495    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2dd459d-f6af-45f3-820d-f57ce3409e29-kube-api-access-jv5wp\" (UniqueName: \"kubernetes.io/projected/c2dd459d-f6af-45f3-820d-f57ce3409e29-kube-api-access-jv5wp\") pod \"c2dd459d-f6af-45f3-820d-f57ce3409e29\" (UID: \"c2dd459d-f6af-45f3-820d-f57ce3409e29\") "
Oct 30 06:18:59.838724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.838701    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2dd459d-f6af-45f3-820d-f57ce3409e29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dd459d-f6af-45f3-820d-f57ce3409e29-plugins-config-volume\") pod \"c2dd459d-f6af-45f3-820d-f57ce3409e29\" (UID: \"c2dd459d-f6af-45f3-820d-f57ce3409e29\") "
Oct 30 06:18:59.839128 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.839101    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dd459d-f6af-45f3-820d-f57ce3409e29-plugins-config-volume" pod "c2dd459d-f6af-45f3-820d-f57ce3409e29" (UID: "c2dd459d-f6af-45f3-820d-f57ce3409e29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:18:59.839398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.839379    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2dd459d-f6af-45f3-820d-f57ce3409e29-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:59.840820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.840793    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dd459d-f6af-45f3-820d-f57ce3409e29-kube-api-access-jv5wp" pod "c2dd459d-f6af-45f3-820d-f57ce3409e29" (UID: "c2dd459d-f6af-45f3-820d-f57ce3409e29"). InnerVolumeSpecName "kube-api-access-jv5wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:18:59.940218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.940109    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fda6e76-0dd9-4775-99ad-9242db7d63cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda6e76-0dd9-4775-99ad-9242db7d63cd-plugins-config-volume\") pod \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\" (UID: \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\") "
Oct 30 06:18:59.940218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.940211    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fda6e76-0dd9-4775-99ad-9242db7d63cd-kube-api-access-4fp7t\" (UniqueName: \"kubernetes.io/projected/4fda6e76-0dd9-4775-99ad-9242db7d63cd-kube-api-access-4fp7t\") pod \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\" (UID: \"4fda6e76-0dd9-4775-99ad-9242db7d63cd\") "
Oct 30 06:18:59.940535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.940507    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fda6e76-0dd9-4775-99ad-9242db7d63cd-plugins-config-volume" pod "4fda6e76-0dd9-4775-99ad-9242db7d63cd" (UID: "4fda6e76-0dd9-4775-99ad-9242db7d63cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:18:59.940958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.940937    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv5wp\" (UniqueName: \"kubernetes.io/projected/c2dd459d-f6af-45f3-820d-f57ce3409e29-kube-api-access-jv5wp\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:59.941039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.940967    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda6e76-0dd9-4775-99ad-9242db7d63cd-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:18:59.942846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.942823    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fda6e76-0dd9-4775-99ad-9242db7d63cd-kube-api-access-4fp7t" pod "4fda6e76-0dd9-4775-99ad-9242db7d63cd" (UID: "4fda6e76-0dd9-4775-99ad-9242db7d63cd"). InnerVolumeSpecName "kube-api-access-4fp7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:18:59.948329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.948307    5250 generic.go:356] "Generic (PLEG): container finished" podID="c2dd459d-f6af-45f3-820d-f57ce3409e29" containerID="fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c" exitCode=0
Oct 30 06:18:59.948430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.948368    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"
Oct 30 06:18:59.948430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.948388    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg" event={"ID":"c2dd459d-f6af-45f3-820d-f57ce3409e29","Type":"ContainerDied","Data":"fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c"}
Oct 30 06:18:59.948543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.948443    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg" event={"ID":"c2dd459d-f6af-45f3-820d-f57ce3409e29","Type":"ContainerDied","Data":"890d6ff9b24619b1dd95684a564af8becd2cbc48f73987588c9ff578b3f78cf9"}
Oct 30 06:18:59.948543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.948471    5250 scope.go:122] "RemoveContainer" containerID="fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c"
Oct 30 06:18:59.949997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.949977    5250 generic.go:356] "Generic (PLEG): container finished" podID="4fda6e76-0dd9-4775-99ad-9242db7d63cd" containerID="a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a" exitCode=0
Oct 30 06:18:59.950123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.950024    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"
Oct 30 06:18:59.950123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.950033    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx" event={"ID":"4fda6e76-0dd9-4775-99ad-9242db7d63cd","Type":"ContainerDied","Data":"a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a"}
Oct 30 06:18:59.950123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.950072    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx" event={"ID":"4fda6e76-0dd9-4775-99ad-9242db7d63cd","Type":"ContainerDied","Data":"571c9286b0e4abc19b39474315965fcdb0e382845144057b4814d2083c1e155d"}
Oct 30 06:18:59.962694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.962665    5250 scope.go:122] "RemoveContainer" containerID="fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c"
Oct 30 06:18:59.966531 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:18:59.966502    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c\": container with ID starting with fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c not found: ID does not exist" containerID="fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c"
Oct 30 06:18:59.966660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.966543    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c"} err="failed to get container status \"fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c\": rpc error: code = NotFound desc = could not find container \"fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c\": container with ID starting with fdd569c668d48c7ac255188a8ede93da17f0ed93ba3b4cd94672d88c843fe16c not found: ID does not exist"
Oct 30 06:18:59.966660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.966568    5250 scope.go:122] "RemoveContainer" containerID="a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a"
Oct 30 06:18:59.979559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.979538    5250 scope.go:122] "RemoveContainer" containerID="a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a"
Oct 30 06:18:59.979834 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:18:59.979818    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a\": container with ID starting with a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a not found: ID does not exist" containerID="a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a"
Oct 30 06:18:59.979882 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:18:59.979842    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a"} err="failed to get container status \"a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a\": rpc error: code = NotFound desc = could not find container \"a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a\": container with ID starting with a52cc4c7f240a41fa74601bda4c26cbd9d847d5152cdc11ce52baf1b62348f8a not found: ID does not exist"
Oct 30 06:19:00.009087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:00.009061    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"]
Oct 30 06:19:00.013311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:00.013289    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37bbcc-7b554884bb-5l9lg"]
Oct 30 06:19:00.042153 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:00.042116    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fp7t\" (UniqueName: \"kubernetes.io/projected/4fda6e76-0dd9-4775-99ad-9242db7d63cd-kube-api-access-4fp7t\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:00.048593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:00.048573    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"]
Oct 30 06:19:00.052382 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:00.052361    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37bbcc-64c7787647-q8wjx"]
Oct 30 06:19:01.260536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.260501    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"]
Oct 30 06:19:01.261427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261403    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4d77e321-e8ce-49e5-98e0-34f236d8cfa1" containerName="revoke-keys"
Oct 30 06:19:01.261427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261426    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d77e321-e8ce-49e5-98e0-34f236d8cfa1" containerName="revoke-keys"
Oct 30 06:19:01.261617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261463    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c2dd459d-f6af-45f3-820d-f57ce3409e29" containerName="payload-processing"
Oct 30 06:19:01.261617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261471    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2dd459d-f6af-45f3-820d-f57ce3409e29" containerName="payload-processing"
Oct 30 06:19:01.261617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261497    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6ea4e14a-9d8b-4460-8bfb-21f904305667" containerName="maas-api"
Oct 30 06:19:01.261617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261505    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea4e14a-9d8b-4460-8bfb-21f904305667" containerName="maas-api"
Oct 30 06:19:01.261617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261553    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4fda6e76-0dd9-4775-99ad-9242db7d63cd" containerName="payload-pre-processing"
Oct 30 06:19:01.261617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261561    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fda6e76-0dd9-4775-99ad-9242db7d63cd" containerName="payload-pre-processing"
Oct 30 06:19:01.261950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261729    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fda6e76-0dd9-4775-99ad-9242db7d63cd" containerName="payload-pre-processing"
Oct 30 06:19:01.261950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261747    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d77e321-e8ce-49e5-98e0-34f236d8cfa1" containerName="revoke-keys"
Oct 30 06:19:01.261950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261764    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2dd459d-f6af-45f3-820d-f57ce3409e29" containerName="payload-processing"
Oct 30 06:19:01.261950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.261776    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea4e14a-9d8b-4460-8bfb-21f904305667" containerName="maas-api"
Oct 30 06:19:01.265663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.265643    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.287012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.286946    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fda6e76-0dd9-4775-99ad-9242db7d63cd" path="/var/lib/kubelet/pods/4fda6e76-0dd9-4775-99ad-9242db7d63cd/volumes"
Oct 30 06:19:01.287616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.287593    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2dd459d-f6af-45f3-820d-f57ce3409e29" path="/var/lib/kubelet/pods/c2dd459d-f6af-45f3-820d-f57ce3409e29/volumes"
Oct 30 06:19:01.287746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.287728    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:19:01.287873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.287855    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:19:01.287964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.287949    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"]
Oct 30 06:19:01.327997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.327968    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:19:01.461420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461381    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-podinfo\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.461625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461467    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-token\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-token\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.461625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461506    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istiod-ca-cert\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.461763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461644    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-kube-api-access-l4jwj\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-kube-api-access-l4jwj\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.461763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461754    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-credential-socket\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.461877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461833    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-socket\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.461940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461875    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-envoy\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.461940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461910    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-data\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.462053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.461975    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-certs\") pod \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\" (UID: \"2f40cc9b-5c8e-414e-8b57-223c8b18b40f\") "
Oct 30 06:19:01.462113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.462038    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-credential-socket" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:01.462171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.462129    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istiod-ca-cert" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:19:01.462362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.462339    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.462446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.462422    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-socket" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:01.462657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.462635    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.462804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.462656    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-certs" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:01.462868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.462772    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvch\" (UniqueName: \"kubernetes.io/projected/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-kube-api-access-jqvch\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.462868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.462846    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-data" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:01.463030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.463005    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.463108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.463026    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.463108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.463043    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.463108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.463058    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.463108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.463070    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.464620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.464597    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-token" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:01.464815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.464789    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-podinfo" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:19:01.465077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.465062    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-kube-api-access-l4jwj" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "kube-api-access-l4jwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:01.465774 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.465752    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-envoy" pod "2f40cc9b-5c8e-414e-8b57-223c8b18b40f" (UID: "2f40cc9b-5c8e-414e-8b57-223c8b18b40f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:01.482651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.482629    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"]
Oct 30 06:19:01.509230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.509195    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"]
Oct 30 06:19:01.564794 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.564713    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.564978 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.564916    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.565050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.564978    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvch\" (UniqueName: \"kubernetes.io/projected/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-kube-api-access-jqvch\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.565163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.565145    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.565294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.565169    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.565294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.565209    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4jwj\" (UniqueName: \"kubernetes.io/projected/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-kube-api-access-l4jwj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.565294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.565225    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f40cc9b-5c8e-414e-8b57-223c8b18b40f-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:01.565970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.565943    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.567765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.567739    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.588151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.588096    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvch\" (UniqueName: \"kubernetes.io/projected/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-kube-api-access-jqvch\") pod \"maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.638813 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.638773    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:01.779356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.779319    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:01.882515 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.882422    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:01.920951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.920916    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:01.975542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.975509    5250 generic.go:356] "Generic (PLEG): container finished" podID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerID="049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e" exitCode=0
Oct 30 06:19:01.975746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.975589    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"
Oct 30 06:19:01.975746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.975592    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" event={"ID":"2f40cc9b-5c8e-414e-8b57-223c8b18b40f","Type":"ContainerDied","Data":"049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e"}
Oct 30 06:19:01.975746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.975649    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx" event={"ID":"2f40cc9b-5c8e-414e-8b57-223c8b18b40f","Type":"ContainerDied","Data":"16de5b094d0fa0ac459376507549134e08d52692a6058a7078f82fbfa28b3f75"}
Oct 30 06:19:01.975746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.975677    5250 scope.go:122] "RemoveContainer" containerID="049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e"
Oct 30 06:19:01.990018 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:01.989892    5250 scope.go:122] "RemoveContainer" containerID="e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e"
Oct 30 06:19:02.152563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.121589    5250 scope.go:122] "RemoveContainer" containerID="049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e"
Oct 30 06:19:02.152563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.146861    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"]
Oct 30 06:19:02.154016 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:19:02.153982    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e\": container with ID starting with 049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e not found: ID does not exist" containerID="049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e"
Oct 30 06:19:02.154159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.154027    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e"} err="failed to get container status \"049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e\": rpc error: code = NotFound desc = could not find container \"049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e\": container with ID starting with 049610072a3ee733b5f85a700f72de771abde2684609a0582b81323b3370e32e not found: ID does not exist"
Oct 30 06:19:02.154159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.154058    5250 scope.go:122] "RemoveContainer" containerID="e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e"
Oct 30 06:19:02.154692 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:19:02.154624    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e\": container with ID starting with e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e not found: ID does not exist" containerID="e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e"
Oct 30 06:19:02.154692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.154656    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e"} err="failed to get container status \"e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e\": rpc error: code = NotFound desc = could not find container \"e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e\": container with ID starting with e2cfb1f78851f4e99bf6fcf837d9cd5a723519b83f21d44759f3553eefa90c7e not found: ID does not exist"
Oct 30 06:19:02.155411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.155388    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ebe61e-openshift-default-6cf8876c95-5k8tx"]
Oct 30 06:19:02.213263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.213232    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"]
Oct 30 06:19:02.215459 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:19:02.215434    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29eaca0_1c74_47c8_a309_6c2e3ccddeea.slice/crio-fcc690562e37778106d96be02cf80d8f1d59b5f38fb99b45f1af80ccde5dc377 WatchSource:0}: Error finding container fcc690562e37778106d96be02cf80d8f1d59b5f38fb99b45f1af80ccde5dc377: Status 404 returned error can't find the container with id fcc690562e37778106d96be02cf80d8f1d59b5f38fb99b45f1af80ccde5dc377
Oct 30 06:19:02.376100 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:19:02.376062    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:19:02.985531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.985486    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww" event={"ID":"d29eaca0-1c74-47c8-a309-6c2e3ccddeea","Type":"ContainerStarted","Data":"b44274c6c27feced8964bc6c6eeefcd69526435a567173abb20aea764960f8f1"}
Oct 30 06:19:02.985749 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:02.985637    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww" event={"ID":"d29eaca0-1c74-47c8-a309-6c2e3ccddeea","Type":"ContainerStarted","Data":"fcc690562e37778106d96be02cf80d8f1d59b5f38fb99b45f1af80ccde5dc377"}
Oct 30 06:19:03.024487 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:03.024436    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww" podStartSLOduration=2.024423242 podStartE2EDuration="2.024423242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:03.021778808 +0000 UTC m=+2857.852816029" watchObservedRunningTime="2026-10-30 06:19:03.024423242 +0000 UTC m=+2857.855460484"
Oct 30 06:19:03.282348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:03.282269    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" path="/var/lib/kubelet/pods/2f40cc9b-5c8e-414e-8b57-223c8b18b40f/volumes"
Oct 30 06:19:03.498639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:03.498606    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:03.996657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:03.996624    5250 generic.go:356] "Generic (PLEG): container finished" podID="d29eaca0-1c74-47c8-a309-6c2e3ccddeea" containerID="b44274c6c27feced8964bc6c6eeefcd69526435a567173abb20aea764960f8f1" exitCode=0
Oct 30 06:19:03.996850 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:03.996713    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww" event={"ID":"d29eaca0-1c74-47c8-a309-6c2e3ccddeea","Type":"ContainerDied","Data":"b44274c6c27feced8964bc6c6eeefcd69526435a567173abb20aea764960f8f1"}
Oct 30 06:19:05.181462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.181434    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:05.227931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.227897    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:05.331503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.331425    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-xks-ca\") pod \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") "
Oct 30 06:19:05.331647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.331504    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-openshift-ca\") pod \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") "
Oct 30 06:19:05.331647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.331526    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-kube-api-access-jqvch\" (UniqueName: \"kubernetes.io/projected/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-kube-api-access-jqvch\") pod \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\" (UID: \"d29eaca0-1c74-47c8-a309-6c2e3ccddeea\") "
Oct 30 06:19:05.331907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.331884    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-openshift-ca" pod "d29eaca0-1c74-47c8-a309-6c2e3ccddeea" (UID: "d29eaca0-1c74-47c8-a309-6c2e3ccddeea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:19:05.332271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.332243    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:05.333887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.333861    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-kube-api-access-jqvch" pod "d29eaca0-1c74-47c8-a309-6c2e3ccddeea" (UID: "d29eaca0-1c74-47c8-a309-6c2e3ccddeea"). InnerVolumeSpecName "kube-api-access-jqvch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:05.333947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.333889    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-xks-ca" pod "d29eaca0-1c74-47c8-a309-6c2e3ccddeea" (UID: "d29eaca0-1c74-47c8-a309-6c2e3ccddeea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:19:05.433513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.433479    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:05.433513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:05.433510    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvch\" (UniqueName: \"kubernetes.io/projected/d29eaca0-1c74-47c8-a309-6c2e3ccddeea-kube-api-access-jqvch\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:06.015395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.015359    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww" event={"ID":"d29eaca0-1c74-47c8-a309-6c2e3ccddeea","Type":"ContainerDied","Data":"fcc690562e37778106d96be02cf80d8f1d59b5f38fb99b45f1af80ccde5dc377"}
Oct 30 06:19:06.015581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.015397    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"
Oct 30 06:19:06.015581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.015427    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcc690562e37778106d96be02cf80d8f1d59b5f38fb99b45f1af80ccde5dc377"
Oct 30 06:19:06.483096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.483056    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz" podUID="07a44d46-bade-4663-98c1-d1e49bb857c7" containerName="payload-processing" containerID="cri-o://922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a" gracePeriod=25
Oct 30 06:19:06.509971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.509935    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg" podUID="108dde7f-dc08-4fe4-8ae8-dad940d32666" containerName="payload-pre-processing" containerID="cri-o://b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f" gracePeriod=25
Oct 30 06:19:06.756405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.749484    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:06.812033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.812006    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:19:06.853458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.853430    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:19:06.853928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.853897    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a44d46-bade-4663-98c1-d1e49bb857c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a44d46-bade-4663-98c1-d1e49bb857c7-plugins-config-volume\") pod \"07a44d46-bade-4663-98c1-d1e49bb857c7\" (UID: \"07a44d46-bade-4663-98c1-d1e49bb857c7\") "
Oct 30 06:19:06.854261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.854237    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a44d46-bade-4663-98c1-d1e49bb857c7-kube-api-access-jpqkm\" (UniqueName: \"kubernetes.io/projected/07a44d46-bade-4663-98c1-d1e49bb857c7-kube-api-access-jpqkm\") pod \"07a44d46-bade-4663-98c1-d1e49bb857c7\" (UID: \"07a44d46-bade-4663-98c1-d1e49bb857c7\") "
Oct 30 06:19:06.854430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.854405    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a44d46-bade-4663-98c1-d1e49bb857c7-plugins-config-volume" pod "07a44d46-bade-4663-98c1-d1e49bb857c7" (UID: "07a44d46-bade-4663-98c1-d1e49bb857c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:19:06.854784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.854755    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a44d46-bade-4663-98c1-d1e49bb857c7-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:06.856615 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.856593    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a44d46-bade-4663-98c1-d1e49bb857c7-kube-api-access-jpqkm" pod "07a44d46-bade-4663-98c1-d1e49bb857c7" (UID: "07a44d46-bade-4663-98c1-d1e49bb857c7"). InnerVolumeSpecName "kube-api-access-jpqkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:06.955725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.955693    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108dde7f-dc08-4fe4-8ae8-dad940d32666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108dde7f-dc08-4fe4-8ae8-dad940d32666-plugins-config-volume\") pod \"108dde7f-dc08-4fe4-8ae8-dad940d32666\" (UID: \"108dde7f-dc08-4fe4-8ae8-dad940d32666\") "
Oct 30 06:19:06.955922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.955740    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108dde7f-dc08-4fe4-8ae8-dad940d32666-kube-api-access-9k7vb\" (UniqueName: \"kubernetes.io/projected/108dde7f-dc08-4fe4-8ae8-dad940d32666-kube-api-access-9k7vb\") pod \"108dde7f-dc08-4fe4-8ae8-dad940d32666\" (UID: \"108dde7f-dc08-4fe4-8ae8-dad940d32666\") "
Oct 30 06:19:06.956211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.956157    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108dde7f-dc08-4fe4-8ae8-dad940d32666-plugins-config-volume" pod "108dde7f-dc08-4fe4-8ae8-dad940d32666" (UID: "108dde7f-dc08-4fe4-8ae8-dad940d32666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:19:06.956491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.956473    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqkm\" (UniqueName: \"kubernetes.io/projected/07a44d46-bade-4663-98c1-d1e49bb857c7-kube-api-access-jpqkm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:06.956536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.956495    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108dde7f-dc08-4fe4-8ae8-dad940d32666-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:06.958189 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:06.958161    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108dde7f-dc08-4fe4-8ae8-dad940d32666-kube-api-access-9k7vb" pod "108dde7f-dc08-4fe4-8ae8-dad940d32666" (UID: "108dde7f-dc08-4fe4-8ae8-dad940d32666"). InnerVolumeSpecName "kube-api-access-9k7vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:07.025215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.025092    5250 generic.go:356] "Generic (PLEG): container finished" podID="07a44d46-bade-4663-98c1-d1e49bb857c7" containerID="922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a" exitCode=0
Oct 30 06:19:07.025215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.025144    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"
Oct 30 06:19:07.025446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.025215    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz" event={"ID":"07a44d46-bade-4663-98c1-d1e49bb857c7","Type":"ContainerDied","Data":"922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a"}
Oct 30 06:19:07.025446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.025263    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz" event={"ID":"07a44d46-bade-4663-98c1-d1e49bb857c7","Type":"ContainerDied","Data":"6367b6cfa50e9c1f126fc96d8124121575bdc4364c8091786934498642438fb6"}
Oct 30 06:19:07.025446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.025295    5250 scope.go:122] "RemoveContainer" containerID="922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a"
Oct 30 06:19:07.026825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.026801    5250 generic.go:356] "Generic (PLEG): container finished" podID="108dde7f-dc08-4fe4-8ae8-dad940d32666" containerID="b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f" exitCode=0
Oct 30 06:19:07.026963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.026852    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"
Oct 30 06:19:07.026963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.026941    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg" event={"ID":"108dde7f-dc08-4fe4-8ae8-dad940d32666","Type":"ContainerDied","Data":"b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f"}
Oct 30 06:19:07.027067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.026990    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg" event={"ID":"108dde7f-dc08-4fe4-8ae8-dad940d32666","Type":"ContainerDied","Data":"68cebbb0bd799f379541ffe3fb559655182419821218e4af54da438d0cb15d21"}
Oct 30 06:19:07.054240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.054211    5250 scope.go:122] "RemoveContainer" containerID="922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a"
Oct 30 06:19:07.055595 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:19:07.055560    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a\": container with ID starting with 922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a not found: ID does not exist" containerID="922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a"
Oct 30 06:19:07.055717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.055605    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a"} err="failed to get container status \"922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a\": rpc error: code = NotFound desc = could not find container \"922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a\": container with ID starting with 922c5bd1f2e317ec3cfcc90433a92ae710cbbe5505bbe80d6ff1501e3b94212a not found: ID does not exist"
Oct 30 06:19:07.055717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.055629    5250 scope.go:122] "RemoveContainer" containerID="b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f"
Oct 30 06:19:07.057832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.057803    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k7vb\" (UniqueName: \"kubernetes.io/projected/108dde7f-dc08-4fe4-8ae8-dad940d32666-kube-api-access-9k7vb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:07.068071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.068050    5250 scope.go:122] "RemoveContainer" containerID="b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f"
Oct 30 06:19:07.068403 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:19:07.068381    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f\": container with ID starting with b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f not found: ID does not exist" containerID="b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f"
Oct 30 06:19:07.068476 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.068413    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f"} err="failed to get container status \"b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f\": rpc error: code = NotFound desc = could not find container \"b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f\": container with ID starting with b8233f64160ec002e157e8b0cc1121dc367211a0643a7ceaff19377cd267449f not found: ID does not exist"
Oct 30 06:19:07.089921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.089895    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"]
Oct 30 06:19:07.094683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.094658    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-56fd999f6b-fkhmz"]
Oct 30 06:19:07.124985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.124954    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"]
Oct 30 06:19:07.130942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.130915    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-c6894d747-ch6wg"]
Oct 30 06:19:07.282145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.282067    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a44d46-bade-4663-98c1-d1e49bb857c7" path="/var/lib/kubelet/pods/07a44d46-bade-4663-98c1-d1e49bb857c7/volumes"
Oct 30 06:19:07.282559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.282545    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108dde7f-dc08-4fe4-8ae8-dad940d32666" path="/var/lib/kubelet/pods/108dde7f-dc08-4fe4-8ae8-dad940d32666/volumes"
Oct 30 06:19:07.922556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:07.922519    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:08.085641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:08.085605    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:09.296643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:09.296605    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:10.906694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:10.906655    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:12.866789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:12.866738    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:14.491678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:14.491639    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:15.977624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:15.977579    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:17.541475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:17.541440    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:18.533740 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:18.533704    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:19.015014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:19.014979    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:20.479261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:20.479212    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:20.624612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:20.624581    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:22.293483 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:22.293432    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:22.892922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:22.892886    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.317567    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"]
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318518    5250 cpu_manager.go:414] "containerMap: removing container" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318538    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318555    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d29eaca0-1c74-47c8-a309-6c2e3ccddeea" containerName="revoke-keys"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318564    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29eaca0-1c74-47c8-a309-6c2e3ccddeea" containerName="revoke-keys"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318583    5250 cpu_manager.go:414] "containerMap: removing container" podUID="07a44d46-bade-4663-98c1-d1e49bb857c7" containerName="payload-processing"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318590    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a44d46-bade-4663-98c1-d1e49bb857c7" containerName="payload-processing"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318665    5250 cpu_manager.go:414] "containerMap: removing container" podUID="108dde7f-dc08-4fe4-8ae8-dad940d32666" containerName="payload-pre-processing"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318672    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108dde7f-dc08-4fe4-8ae8-dad940d32666" containerName="payload-pre-processing"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318829    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318840    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="108dde7f-dc08-4fe4-8ae8-dad940d32666" containerName="payload-pre-processing"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318855    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318865    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29eaca0-1c74-47c8-a309-6c2e3ccddeea" containerName="revoke-keys"
Oct 30 06:19:23.320256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.318875    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a44d46-bade-4663-98c1-d1e49bb857c7" containerName="payload-processing"
Oct 30 06:19:23.355376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.355347    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.409267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.399163    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d4bace-openshift-default-dockercfg-lrjgx\""
Oct 30 06:19:23.410219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.410009    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:23.413939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.413916    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"]
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.447519    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-token\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.500004    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istiod-ca-cert\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.500053    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-credential-socket\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.500113    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-data\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.500325    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qsrq\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-kube-api-access-7qsrq\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.500396    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-certs\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.500437    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-podinfo\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.500600    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-envoy\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.504205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.500665    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-socket\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602210    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-credential-socket\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602264    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-data\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602351    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qsrq\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-kube-api-access-7qsrq\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602391    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-certs\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602440    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-podinfo\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602555    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-envoy\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602599    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-credential-socket\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602614    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-socket\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602648    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-data\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.602721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602696    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-certs\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.603011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602759    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-token\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.603011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602831    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istiod-ca-cert\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.603011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.602910    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-socket\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.603542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.603478    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istiod-ca-cert\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.605075 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.605051    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-envoy\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.605493 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.605467    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-podinfo\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.624367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.624327    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-token\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.625120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.625088    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qsrq\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-kube-api-access-7qsrq\") pod \"e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") " pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.814467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.814431    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:23.894100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:23.894027    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:24.041952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:24.041917    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:24.146113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:24.146088    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"]
Oct 30 06:19:24.148408 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:19:24.148380    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674faa9d_8c41_44bd_8d47_07c8c671fdc5.slice/crio-091f86378d2fc7114815061adb484432b30fc301f8df955f396a3176cf40a0f0 WatchSource:0}: Error finding container 091f86378d2fc7114815061adb484432b30fc301f8df955f396a3176cf40a0f0: Status 404 returned error can't find the container with id 091f86378d2fc7114815061adb484432b30fc301f8df955f396a3176cf40a0f0
Oct 30 06:19:24.151273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:24.151239    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:19:24.151381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:24.151307    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:19:24.151381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:24.151337    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:19:24.235804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:24.235769    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" event={"ID":"674faa9d-8c41-44bd-8d47-07c8c671fdc5","Type":"ContainerStarted","Data":"091f86378d2fc7114815061adb484432b30fc301f8df955f396a3176cf40a0f0"}
Oct 30 06:19:24.279971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:24.279705    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" podStartSLOduration=1.279690318 podStartE2EDuration="1.279690318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:24.277219763 +0000 UTC m=+2879.108256981" watchObservedRunningTime="2026-10-30 06:19:24.279690318 +0000 UTC m=+2879.110727534"
Oct 30 06:19:24.814890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:24.814856    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:25.246038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.246003    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" event={"ID":"674faa9d-8c41-44bd-8d47-07c8c671fdc5","Type":"ContainerStarted","Data":"19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565"}
Oct 30 06:19:25.296004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.295978    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"]
Oct 30 06:19:25.296674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.296658    5250 cpu_manager.go:414] "containerMap: removing container" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy"
Oct 30 06:19:25.296745 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.296676    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f40cc9b-5c8e-414e-8b57-223c8b18b40f" containerName="istio-proxy"
Oct 30 06:19:25.300561 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.300542    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.323569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.323548    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:19:25.323756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.323565    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:19:25.330538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.330510    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"]
Oct 30 06:19:25.431161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.431128    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fmf5\" (UniqueName: \"kubernetes.io/projected/e9722ef4-cade-41cb-a8e1-a70ca066bb35-kube-api-access-4fmf5\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.431364 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.431202    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.431457 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.431441    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.533241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.533143    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmf5\" (UniqueName: \"kubernetes.io/projected/e9722ef4-cade-41cb-a8e1-a70ca066bb35-kube-api-access-4fmf5\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.533241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.533231    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.533547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.533523    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.533876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.533858    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.545022 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.544992    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:19:25.554517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.554435    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fmf5\" (UniqueName: \"kubernetes.io/projected/e9722ef4-cade-41cb-a8e1-a70ca066bb35-kube-api-access-4fmf5\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.556053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.556030    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2f94e0-lw8lr\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.627076 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.627045    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:19:25.641113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.637732    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:25.642940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.641331    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:25.821072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.821002    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:25.961323 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:25.961293    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"]
Oct 30 06:19:25.962970 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:19:25.962945    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9722ef4_cade_41cb_a8e1_a70ca066bb35.slice/crio-91c405e68453f6ba71f998a03974ec0c87760324f402823539cc56b00985576f WatchSource:0}: Error finding container 91c405e68453f6ba71f998a03974ec0c87760324f402823539cc56b00985576f: Status 404 returned error can't find the container with id 91c405e68453f6ba71f998a03974ec0c87760324f402823539cc56b00985576f
Oct 30 06:19:26.257171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:26.257137    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr" event={"ID":"e9722ef4-cade-41cb-a8e1-a70ca066bb35","Type":"ContainerStarted","Data":"91c405e68453f6ba71f998a03974ec0c87760324f402823539cc56b00985576f"}
Oct 30 06:19:26.257417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:26.257320    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:26.258378 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:26.258359    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:19:26.635078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:26.635039    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:26.786949 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:26.786918    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:27.277107 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:27.277018    5250 generic.go:356] "Generic (PLEG): container finished" podID="e9722ef4-cade-41cb-a8e1-a70ca066bb35" containerID="88a9d85195ff0246d20c83d01dac105f180f139ab0f63905255a4e2d0b1ebfc4" exitCode=0
Oct 30 06:19:27.284261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:27.284231    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr" event={"ID":"e9722ef4-cade-41cb-a8e1-a70ca066bb35","Type":"ContainerDied","Data":"88a9d85195ff0246d20c83d01dac105f180f139ab0f63905255a4e2d0b1ebfc4"}
Oct 30 06:19:27.776974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:27.776941    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:28.207608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.207576    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:28.294558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.294504    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"]
Oct 30 06:19:28.299372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.299351    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:28.323308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.323278    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d4bace-dockercfg-7q2ql\""
Oct 30 06:19:28.323888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.323859    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d4bace\""
Oct 30 06:19:28.331143 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.331114    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"]
Oct 30 06:19:28.353815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.353789    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:28.366429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.366406    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"]
Oct 30 06:19:28.370966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.370939    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:28.392513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.392472    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbm4\" (UniqueName: \"kubernetes.io/projected/6763a628-af89-489c-afd2-a60dfb5370da-kube-api-access-crbm4\") pod \"payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq\" (UID: \"6763a628-af89-489c-afd2-a60dfb5370da\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:28.402569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.402480    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763a628-af89-489c-afd2-a60dfb5370da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq\" (UID: \"6763a628-af89-489c-afd2-a60dfb5370da\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:28.403352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.403319    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"]
Oct 30 06:19:28.531268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.530905    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00655df9-4ba7-4c1b-b23b-b48315f0129d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28\" (UID: \"00655df9-4ba7-4c1b-b23b-b48315f0129d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:28.531268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.530989    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbm4\" (UniqueName: \"kubernetes.io/projected/6763a628-af89-489c-afd2-a60dfb5370da-kube-api-access-crbm4\") pod \"payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq\" (UID: \"6763a628-af89-489c-afd2-a60dfb5370da\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:28.531268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.531084    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mq6\" (UniqueName: \"kubernetes.io/projected/00655df9-4ba7-4c1b-b23b-b48315f0129d-kube-api-access-w9mq6\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28\" (UID: \"00655df9-4ba7-4c1b-b23b-b48315f0129d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:28.607700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.603811    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763a628-af89-489c-afd2-a60dfb5370da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq\" (UID: \"6763a628-af89-489c-afd2-a60dfb5370da\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:28.607700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.605202    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763a628-af89-489c-afd2-a60dfb5370da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq\" (UID: \"6763a628-af89-489c-afd2-a60dfb5370da\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:28.655151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.655096    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbm4\" (UniqueName: \"kubernetes.io/projected/6763a628-af89-489c-afd2-a60dfb5370da-kube-api-access-crbm4\") pod \"payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq\" (UID: \"6763a628-af89-489c-afd2-a60dfb5370da\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:28.690913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.690881    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:28.705127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.705098    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00655df9-4ba7-4c1b-b23b-b48315f0129d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28\" (UID: \"00655df9-4ba7-4c1b-b23b-b48315f0129d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:28.705268 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.705231    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mq6\" (UniqueName: \"kubernetes.io/projected/00655df9-4ba7-4c1b-b23b-b48315f0129d-kube-api-access-w9mq6\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28\" (UID: \"00655df9-4ba7-4c1b-b23b-b48315f0129d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:28.706031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.706006    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00655df9-4ba7-4c1b-b23b-b48315f0129d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28\" (UID: \"00655df9-4ba7-4c1b-b23b-b48315f0129d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:28.726668 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.726643    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mq6\" (UniqueName: \"kubernetes.io/projected/00655df9-4ba7-4c1b-b23b-b48315f0129d-kube-api-access-w9mq6\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28\" (UID: \"00655df9-4ba7-4c1b-b23b-b48315f0129d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:28.806897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.806805    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-xks-ca\") pod \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") "
Oct 30 06:19:28.806897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.806858    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9722ef4-cade-41cb-a8e1-a70ca066bb35-kube-api-access-4fmf5\" (UniqueName: \"kubernetes.io/projected/e9722ef4-cade-41cb-a8e1-a70ca066bb35-kube-api-access-4fmf5\") pod \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") "
Oct 30 06:19:28.807125 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.806950    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-openshift-ca\") pod \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\" (UID: \"e9722ef4-cade-41cb-a8e1-a70ca066bb35\") "
Oct 30 06:19:28.807335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.807310    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-openshift-ca" pod "e9722ef4-cade-41cb-a8e1-a70ca066bb35" (UID: "e9722ef4-cade-41cb-a8e1-a70ca066bb35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:19:28.807724 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.807700    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:28.809094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.809074    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-xks-ca" pod "e9722ef4-cade-41cb-a8e1-a70ca066bb35" (UID: "e9722ef4-cade-41cb-a8e1-a70ca066bb35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:19:28.809173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.809151    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9722ef4-cade-41cb-a8e1-a70ca066bb35-kube-api-access-4fmf5" pod "e9722ef4-cade-41cb-a8e1-a70ca066bb35" (UID: "e9722ef4-cade-41cb-a8e1-a70ca066bb35"). InnerVolumeSpecName "kube-api-access-4fmf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:28.909639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.909597    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9722ef4-cade-41cb-a8e1-a70ca066bb35-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:28.909639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.909632    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fmf5\" (UniqueName: \"kubernetes.io/projected/e9722ef4-cade-41cb-a8e1-a70ca066bb35-kube-api-access-4fmf5\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:28.915051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.915026    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:28.988607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:28.988578    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:29.223481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:29.223452    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"]
Oct 30 06:19:29.224616 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:19:29.224589    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6763a628_af89_489c_afd2_a60dfb5370da.slice/crio-da1af2d318764b839b87ae8c8986cc0b44f1643e9aec238f84c179cd6de4394f WatchSource:0}: Error finding container da1af2d318764b839b87ae8c8986cc0b44f1643e9aec238f84c179cd6de4394f: Status 404 returned error can't find the container with id da1af2d318764b839b87ae8c8986cc0b44f1643e9aec238f84c179cd6de4394f
Oct 30 06:19:29.307361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:29.307322    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq" event={"ID":"6763a628-af89-489c-afd2-a60dfb5370da","Type":"ContainerStarted","Data":"da1af2d318764b839b87ae8c8986cc0b44f1643e9aec238f84c179cd6de4394f"}
Oct 30 06:19:29.309171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:29.309148    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"
Oct 30 06:19:29.309171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:29.309190    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr" event={"ID":"e9722ef4-cade-41cb-a8e1-a70ca066bb35","Type":"ContainerDied","Data":"91c405e68453f6ba71f998a03974ec0c87760324f402823539cc56b00985576f"}
Oct 30 06:19:29.309366 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:29.309243    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c405e68453f6ba71f998a03974ec0c87760324f402823539cc56b00985576f"
Oct 30 06:19:29.325983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:29.325957    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"]
Oct 30 06:19:29.326792 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:19:29.326770    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00655df9_4ba7_4c1b_b23b_b48315f0129d.slice/crio-435a2880b938fcace574a415a2462eb2decbf8dbcc7e69c3b1021786a64776e7 WatchSource:0}: Error finding container 435a2880b938fcace574a415a2462eb2decbf8dbcc7e69c3b1021786a64776e7: Status 404 returned error can't find the container with id 435a2880b938fcace574a415a2462eb2decbf8dbcc7e69c3b1021786a64776e7
Oct 30 06:19:29.674387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:29.674358    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:30.319114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:30.319076    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq" event={"ID":"6763a628-af89-489c-afd2-a60dfb5370da","Type":"ContainerStarted","Data":"fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b"}
Oct 30 06:19:30.319624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:30.319142    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:30.320994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:30.320953    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28" event={"ID":"00655df9-4ba7-4c1b-b23b-b48315f0129d","Type":"ContainerStarted","Data":"11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974"}
Oct 30 06:19:30.321130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:30.321007    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:30.321130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:30.321023    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28" event={"ID":"00655df9-4ba7-4c1b-b23b-b48315f0129d","Type":"ContainerStarted","Data":"435a2880b938fcace574a415a2462eb2decbf8dbcc7e69c3b1021786a64776e7"}
Oct 30 06:19:30.357482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:30.357436    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq" podStartSLOduration=2.357424355 podStartE2EDuration="2.357424355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:30.354591692 +0000 UTC m=+2885.185628908" watchObservedRunningTime="2026-10-30 06:19:30.357424355 +0000 UTC m=+2885.188461571"
Oct 30 06:19:30.391742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:30.391693    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28" podStartSLOduration=2.391680201 podStartE2EDuration="2.391680201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:30.389800589 +0000 UTC m=+2885.220837807" watchObservedRunningTime="2026-10-30 06:19:30.391680201 +0000 UTC m=+2885.222717423"
Oct 30 06:19:31.386912 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:31.386881    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:32.374934 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:19:32.374895    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:19:32.666169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:32.666117    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:33.292649 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:33.292556    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:33.434020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:33.433986    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:33.625110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:33.625017    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:33.911872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:33.911824    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:35.331884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:35.331849    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:36.995723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:36.995599    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:37.146329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:37.144562    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:37.433196 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:37.433138    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:38.032330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:38.032299    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"]
Oct 30 06:19:38.041810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:38.041784    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cdd01b1-wpzrx"]
Oct 30 06:19:39.255959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:39.255919    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:39.341474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:39.341425    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544dc38c-dfe2-4ab3-b3d9-04e44d35f05b" path="/var/lib/kubelet/pods/544dc38c-dfe2-4ab3-b3d9-04e44d35f05b/volumes"
Oct 30 06:19:40.241657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:40.241623    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:40.387672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:40.387638    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:40.514029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:40.513936    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"]
Oct 30 06:19:40.514550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:40.514515    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy" containerID="cri-o://f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea" gracePeriod=30
Oct 30 06:19:40.574439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:40.574402    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:40.827234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:40.827042    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:40.996675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:40.996635    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:41.156167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:41.156117    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:41.332481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:41.332451    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:19:41.332658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:41.332537    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:19:41.909228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:41.909164    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:44.472509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:44.472476    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:45.793762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.793696    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 06:19:45.898517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898484    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-data\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.898708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898547    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-podinfo\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.898708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898570    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-credential-socket\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.898708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898688    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-socket\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.898888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898723    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-envoy\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.898888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898778    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istiod-ca-cert\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.898888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898816    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-data" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:45.898888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898834    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-token\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-token\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.898888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898862    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-kube-api-access-c68xm\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-kube-api-access-c68xm\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.899212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.898960    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-certs\") pod \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\" (UID: \"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6\") "
Oct 30 06:19:45.899212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.899037    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-credential-socket" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:45.899915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.899810    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istiod-ca-cert" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:19:45.899915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.899884    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:45.899915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.899907    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:45.900161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.899974    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-socket" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:45.900161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.900015    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-certs" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:45.901412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.901387    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-envoy" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:19:45.901565 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.901549    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-kube-api-access-c68xm" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "kube-api-access-c68xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:45.901813 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.901797    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-podinfo" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:19:45.903328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.903308    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-token" pod "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" (UID: "cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:45.982642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.982610    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"]
Oct 30 06:19:45.988235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:45.988201    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"]
Oct 30 06:19:46.001815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.001775    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:46.001815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.001804    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:46.001815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.001819    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:46.002048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.001832    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:46.002048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.001843    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:46.002048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.001854    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c68xm\" (UniqueName: \"kubernetes.io/projected/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-kube-api-access-c68xm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:46.002048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.001866    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:46.138843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.138805    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:46.288038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.288005    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:46.472319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.472226    5250 generic.go:356] "Generic (PLEG): container finished" podID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerID="f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea" exitCode=0
Oct 30 06:19:46.472319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.472280    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"
Oct 30 06:19:46.472319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.472296    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" event={"ID":"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6","Type":"ContainerDied","Data":"f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea"}
Oct 30 06:19:46.472624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.472374    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p" event={"ID":"cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6","Type":"ContainerDied","Data":"1563dfa24bd01ace8449932993c90cf4c177dc73d74b217f7788c5d050ef2b63"}
Oct 30 06:19:46.472624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.472403    5250 scope.go:122] "RemoveContainer" containerID="f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea"
Oct 30 06:19:46.485538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.485520    5250 scope.go:122] "RemoveContainer" containerID="f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea"
Oct 30 06:19:46.485819 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:19:46.485801    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea\": container with ID starting with f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea not found: ID does not exist" containerID="f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea"
Oct 30 06:19:46.485869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.485827    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea"} err="failed to get container status \"f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea\": rpc error: code = NotFound desc = could not find container \"f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea\": container with ID starting with f733c8c01b92b53db42073afdfb8171402399a3c69e5287c47c6ef361b824eea not found: ID does not exist"
Oct 30 06:19:46.520959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.520932    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"]
Oct 30 06:19:46.530321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:46.530297    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-g5g4p"]
Oct 30 06:19:47.282433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:47.282390    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" path="/var/lib/kubelet/pods/cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6/volumes"
Oct 30 06:19:48.000127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:48.000090    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:49.031326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:49.031294    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"]
Oct 30 06:19:49.041191 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:49.041155    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388194bc-txnpz"]
Oct 30 06:19:49.282041 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:49.281964    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cb89ee-32a9-4a1e-a8c9-df1c7e9000af" path="/var/lib/kubelet/pods/89cb89ee-32a9-4a1e-a8c9-df1c7e9000af/volumes"
Oct 30 06:19:49.591570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:49.591481    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:51.585092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:51.585053    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:51.728145 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:51.728108    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:51.878883 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:51.878795    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:52.954608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:52.954567    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:53.187301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.187259    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:53.339064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.338966    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:53.523101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.523054    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"]
Oct 30 06:19:53.524611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.524586    5250 cpu_manager.go:414] "containerMap: removing container" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy"
Oct 30 06:19:53.524779 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.524764    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy"
Oct 30 06:19:53.524895 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.524882    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e9722ef4-cade-41cb-a8e1-a70ca066bb35" containerName="revoke-keys"
Oct 30 06:19:53.524979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.524967    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9722ef4-cade-41cb-a8e1-a70ca066bb35" containerName="revoke-keys"
Oct 30 06:19:53.525330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.525313    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9722ef4-cade-41cb-a8e1-a70ca066bb35" containerName="revoke-keys"
Oct 30 06:19:53.525471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.525459    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd77eb20-958f-4ff7-b4c5-d0ed079b3ff6" containerName="istio-proxy"
Oct 30 06:19:53.534546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.534500    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"]
Oct 30 06:19:53.534828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.534809    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.559390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.557971    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:19:53.559390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.558385    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:53.567142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.567108    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:19:53.629777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.629693    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.629947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.629818    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jh6f\" (UniqueName: \"kubernetes.io/projected/ea7617e8-53a7-4240-9cf3-03aceedc9153-kube-api-access-8jh6f\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.630034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.630018    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.731911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.731875    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.732118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.731967    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jh6f\" (UniqueName: \"kubernetes.io/projected/ea7617e8-53a7-4240-9cf3-03aceedc9153-kube-api-access-8jh6f\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.732219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.732150    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.732729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.732704    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.735130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.735102    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.753959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.753937    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jh6f\" (UniqueName: \"kubernetes.io/projected/ea7617e8-53a7-4240-9cf3-03aceedc9153-kube-api-access-8jh6f\") pod \"maas-api-revoke-sub-cd40e345-hbcq2\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:53.830622 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.830585    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"]
Oct 30 06:19:53.833085 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.833053    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:53.834851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.834829    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de2bf2-fjjww"]
Oct 30 06:19:53.877337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:53.877302    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:54.021422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:54.021387    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:54.195240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:54.195210    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"]
Oct 30 06:19:54.197194 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:19:54.197147    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7617e8_53a7_4240_9cf3_03aceedc9153.slice/crio-09be2c67916c238c4598af261fe43dc58452b23248a0fb36972b798dbd8e99bb WatchSource:0}: Error finding container 09be2c67916c238c4598af261fe43dc58452b23248a0fb36972b798dbd8e99bb: Status 404 returned error can't find the container with id 09be2c67916c238c4598af261fe43dc58452b23248a0fb36972b798dbd8e99bb
Oct 30 06:19:54.398413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:54.398378    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:54.574754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:54.574726    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2" event={"ID":"ea7617e8-53a7-4240-9cf3-03aceedc9153","Type":"ContainerStarted","Data":"09be2c67916c238c4598af261fe43dc58452b23248a0fb36972b798dbd8e99bb"}
Oct 30 06:19:55.282349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:55.282318    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d29eaca0-1c74-47c8-a309-6c2e3ccddeea" path="/var/lib/kubelet/pods/d29eaca0-1c74-47c8-a309-6c2e3ccddeea/volumes"
Oct 30 06:19:55.584744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:55.584656    5250 generic.go:356] "Generic (PLEG): container finished" podID="ea7617e8-53a7-4240-9cf3-03aceedc9153" containerID="99ad8722db3598147a68e5ad0345c6f2b2138e4fae4afc638cdb7cc68b6070da" exitCode=0
Oct 30 06:19:55.584926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:55.584750    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2" event={"ID":"ea7617e8-53a7-4240-9cf3-03aceedc9153","Type":"ContainerDied","Data":"99ad8722db3598147a68e5ad0345c6f2b2138e4fae4afc638cdb7cc68b6070da"}
Oct 30 06:19:55.802648 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:55.799231    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:56.755672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.755649    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:56.880138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.880049    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-xks-ca\") pod \"ea7617e8-53a7-4240-9cf3-03aceedc9153\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") "
Oct 30 06:19:56.880138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.880110    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7617e8-53a7-4240-9cf3-03aceedc9153-kube-api-access-8jh6f\" (UniqueName: \"kubernetes.io/projected/ea7617e8-53a7-4240-9cf3-03aceedc9153-kube-api-access-8jh6f\") pod \"ea7617e8-53a7-4240-9cf3-03aceedc9153\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") "
Oct 30 06:19:56.880365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.880333    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-openshift-ca\") pod \"ea7617e8-53a7-4240-9cf3-03aceedc9153\" (UID: \"ea7617e8-53a7-4240-9cf3-03aceedc9153\") "
Oct 30 06:19:56.880715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.880687    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-openshift-ca" pod "ea7617e8-53a7-4240-9cf3-03aceedc9153" (UID: "ea7617e8-53a7-4240-9cf3-03aceedc9153"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:19:56.881140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.881120    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:56.882619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.882588    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-xks-ca" pod "ea7617e8-53a7-4240-9cf3-03aceedc9153" (UID: "ea7617e8-53a7-4240-9cf3-03aceedc9153"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:19:56.882725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.882615    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7617e8-53a7-4240-9cf3-03aceedc9153-kube-api-access-8jh6f" pod "ea7617e8-53a7-4240-9cf3-03aceedc9153" (UID: "ea7617e8-53a7-4240-9cf3-03aceedc9153"). InnerVolumeSpecName "kube-api-access-8jh6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:19:56.982352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.982318    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7617e8-53a7-4240-9cf3-03aceedc9153-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:56.982352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:56.982347    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jh6f\" (UniqueName: \"kubernetes.io/projected/ea7617e8-53a7-4240-9cf3-03aceedc9153-kube-api-access-8jh6f\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:19:57.605067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:57.604986    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"
Oct 30 06:19:57.605067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:57.605018    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2" event={"ID":"ea7617e8-53a7-4240-9cf3-03aceedc9153","Type":"ContainerDied","Data":"09be2c67916c238c4598af261fe43dc58452b23248a0fb36972b798dbd8e99bb"}
Oct 30 06:19:57.605349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:57.605066    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09be2c67916c238c4598af261fe43dc58452b23248a0fb36972b798dbd8e99bb"
Oct 30 06:19:57.635425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:57.635392    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:59.269466 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:59.269424    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:59.478991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:59.478953    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:19:59.946217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:19:59.946132    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:00.568075 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:00.568027    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:00.863548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:00.863452    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:01.007203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.007144    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" containerID="cri-o://e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f" gracePeriod=45
Oct 30 06:20:01.016742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.015635    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:01.021973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.017702    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" containerID="cri-o://1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1" gracePeriod=45
Oct 30 06:20:01.140590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.140544    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": dial tcp 10.133.0.83:8000: connect: connection refused"
Oct 30 06:20:01.142657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.142628    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"]
Oct 30 06:20:01.142906 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.142870    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy" containerID="cri-o://d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1" gracePeriod=30
Oct 30 06:20:01.217395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.217353    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:01.549273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.549241    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 06:20:01.552065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.552042    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 06:20:01.615673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615282    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-home\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-home\") pod \"c8824800-8838-4200-8f50-4927e8075d3b\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") "
Oct 30 06:20:01.615673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615422    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-model-cache\") pod \"0c92033e-9264-4ae8-b0b6-779c547dba3c\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") "
Oct 30 06:20:01.615673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615484    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-tmp-dir\") pod \"c8824800-8838-4200-8f50-4927e8075d3b\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") "
Oct 30 06:20:01.615673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615552    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-dshm\") pod \"c8824800-8838-4200-8f50-4927e8075d3b\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") "
Oct 30 06:20:01.615673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615643    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8824800-8838-4200-8f50-4927e8075d3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8824800-8838-4200-8f50-4927e8075d3b-tls-certs\") pod \"c8824800-8838-4200-8f50-4927e8075d3b\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") "
Oct 30 06:20:01.616435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615688    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-tmp-dir\") pod \"0c92033e-9264-4ae8-b0b6-779c547dba3c\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") "
Oct 30 06:20:01.616435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615807    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c92033e-9264-4ae8-b0b6-779c547dba3c-kube-api-access-k2c57\" (UniqueName: \"kubernetes.io/projected/0c92033e-9264-4ae8-b0b6-779c547dba3c-kube-api-access-k2c57\") pod \"0c92033e-9264-4ae8-b0b6-779c547dba3c\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") "
Oct 30 06:20:01.616435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615898    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-home\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-home\") pod \"0c92033e-9264-4ae8-b0b6-779c547dba3c\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") "
Oct 30 06:20:01.616435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615933    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-dshm\") pod \"0c92033e-9264-4ae8-b0b6-779c547dba3c\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") "
Oct 30 06:20:01.616435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.615961    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c92033e-9264-4ae8-b0b6-779c547dba3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c92033e-9264-4ae8-b0b6-779c547dba3c-tls-certs\") pod \"0c92033e-9264-4ae8-b0b6-779c547dba3c\" (UID: \"0c92033e-9264-4ae8-b0b6-779c547dba3c\") "
Oct 30 06:20:01.616435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.616011    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-model-cache\") pod \"c8824800-8838-4200-8f50-4927e8075d3b\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") "
Oct 30 06:20:01.616435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.616088    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8824800-8838-4200-8f50-4927e8075d3b-kube-api-access-xf7sg\" (UniqueName: \"kubernetes.io/projected/c8824800-8838-4200-8f50-4927e8075d3b-kube-api-access-xf7sg\") pod \"c8824800-8838-4200-8f50-4927e8075d3b\" (UID: \"c8824800-8838-4200-8f50-4927e8075d3b\") "
Oct 30 06:20:01.617436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.617398    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-tmp-dir" pod "c8824800-8838-4200-8f50-4927e8075d3b" (UID: "c8824800-8838-4200-8f50-4927e8075d3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:01.617772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.617577    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-home" pod "c8824800-8838-4200-8f50-4927e8075d3b" (UID: "c8824800-8838-4200-8f50-4927e8075d3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:01.617892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.617859    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-model-cache" pod "0c92033e-9264-4ae8-b0b6-779c547dba3c" (UID: "0c92033e-9264-4ae8-b0b6-779c547dba3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:01.618433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.618393    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-home" pod "0c92033e-9264-4ae8-b0b6-779c547dba3c" (UID: "0c92033e-9264-4ae8-b0b6-779c547dba3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:01.620965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.620932    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-tmp-dir" pod "0c92033e-9264-4ae8-b0b6-779c547dba3c" (UID: "0c92033e-9264-4ae8-b0b6-779c547dba3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:01.626524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.621581    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-model-cache" pod "c8824800-8838-4200-8f50-4927e8075d3b" (UID: "c8824800-8838-4200-8f50-4927e8075d3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:01.627412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.627374    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-dshm" pod "0c92033e-9264-4ae8-b0b6-779c547dba3c" (UID: "0c92033e-9264-4ae8-b0b6-779c547dba3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:01.628449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.628424    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8824800-8838-4200-8f50-4927e8075d3b-kube-api-access-xf7sg" pod "c8824800-8838-4200-8f50-4927e8075d3b" (UID: "c8824800-8838-4200-8f50-4927e8075d3b"). InnerVolumeSpecName "kube-api-access-xf7sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:01.628890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.628868    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8824800-8838-4200-8f50-4927e8075d3b-tls-certs" pod "c8824800-8838-4200-8f50-4927e8075d3b" (UID: "c8824800-8838-4200-8f50-4927e8075d3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:20:01.629220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.629158    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c92033e-9264-4ae8-b0b6-779c547dba3c-kube-api-access-k2c57" pod "0c92033e-9264-4ae8-b0b6-779c547dba3c" (UID: "0c92033e-9264-4ae8-b0b6-779c547dba3c"). InnerVolumeSpecName "kube-api-access-k2c57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:01.629468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.629443    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c92033e-9264-4ae8-b0b6-779c547dba3c-tls-certs" pod "0c92033e-9264-4ae8-b0b6-779c547dba3c" (UID: "0c92033e-9264-4ae8-b0b6-779c547dba3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:20:01.629994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.629958    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-dshm" pod "c8824800-8838-4200-8f50-4927e8075d3b" (UID: "c8824800-8838-4200-8f50-4927e8075d3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:01.655072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.655040    5250 generic.go:356] "Generic (PLEG): container finished" podID="c8824800-8838-4200-8f50-4927e8075d3b" containerID="e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f" exitCode=0
Oct 30 06:20:01.655442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.655415    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" event={"ID":"c8824800-8838-4200-8f50-4927e8075d3b","Type":"ContainerDied","Data":"e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f"}
Oct 30 06:20:01.655627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.655609    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk" event={"ID":"c8824800-8838-4200-8f50-4927e8075d3b","Type":"ContainerDied","Data":"13e8f5ba25ec3077f2c318b80011b3427c323a3f436cdb7a5992dc7be8dbdddd"}
Oct 30 06:20:01.655742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.655729    5250 scope.go:122] "RemoveContainer" containerID="e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f"
Oct 30 06:20:01.656498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.656156    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"
Oct 30 06:20:01.659202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.659102    5250 generic.go:356] "Generic (PLEG): container finished" podID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerID="1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1" exitCode=0
Oct 30 06:20:01.660349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.659533    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" event={"ID":"0c92033e-9264-4ae8-b0b6-779c547dba3c","Type":"ContainerDied","Data":"1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1"}
Oct 30 06:20:01.660349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.659574    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" event={"ID":"0c92033e-9264-4ae8-b0b6-779c547dba3c","Type":"ContainerDied","Data":"ff5111655cfa130c792ca6f0622b06ca1cc40382271239c627324c724cd398bc"}
Oct 30 06:20:01.660349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.659634    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"
Oct 30 06:20:01.679376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.676674    5250 scope.go:122] "RemoveContainer" containerID="e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f"
Oct 30 06:20:01.682260 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:01.680204    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f\": container with ID starting with e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f not found: ID does not exist" containerID="e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f"
Oct 30 06:20:01.682260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.680250    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f"} err="failed to get container status \"e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f\": rpc error: code = NotFound desc = could not find container \"e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f\": container with ID starting with e567c7cb5f05740f0fa869cca85f87b34a08cbd23c306a5f6fb7a01a756f4e9f not found: ID does not exist"
Oct 30 06:20:01.682260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.680280    5250 scope.go:122] "RemoveContainer" containerID="1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1"
Oct 30 06:20:01.707828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.707031    5250 scope.go:122] "RemoveContainer" containerID="1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1"
Oct 30 06:20:01.707973 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:01.707915    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1\": container with ID starting with 1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1 not found: ID does not exist" containerID="1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1"
Oct 30 06:20:01.708055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.707961    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1"} err="failed to get container status \"1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1\": rpc error: code = NotFound desc = could not find container \"1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1\": container with ID starting with 1b37fadbda593a3ea16d36ff4ec145d55822d18fb8812270e26d6fba3332a9c1 not found: ID does not exist"
Oct 30 06:20:01.718563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.718529    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.718869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.718850    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8824800-8838-4200-8f50-4927e8075d3b-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.719015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.719003    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.719141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.719129    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2c57\" (UniqueName: \"kubernetes.io/projected/0c92033e-9264-4ae8-b0b6-779c547dba3c-kube-api-access-k2c57\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.719283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.719268    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.719395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.719383    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.719570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.719556    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c92033e-9264-4ae8-b0b6-779c547dba3c-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.719690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.719678    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.719803 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.719791    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7sg\" (UniqueName: \"kubernetes.io/projected/c8824800-8838-4200-8f50-4927e8075d3b-kube-api-access-xf7sg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.719914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.719901    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.720023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.720010    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c92033e-9264-4ae8-b0b6-779c547dba3c-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.720118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.720107    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8824800-8838-4200-8f50-4927e8075d3b-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:01.720237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.718765    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"]
Oct 30 06:20:01.724032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.724010    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-2-w2-e7a506-kserve-595954758d-lvgmk"]
Oct 30 06:20:01.764810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.764712    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"]
Oct 30 06:20:01.770641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:01.770607    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2"]
Oct 30 06:20:02.375037 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:02.374954    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:20:02.490204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:02.487279    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/distinct-w2-e7a506-kserve-554f6d58f5-rt2s2" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:20:03.050066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:03.050028    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:03.281801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:03.281766    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" path="/var/lib/kubelet/pods/0c92033e-9264-4ae8-b0b6-779c547dba3c/volumes"
Oct 30 06:20:03.282318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:03.282300    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8824800-8838-4200-8f50-4927e8075d3b" path="/var/lib/kubelet/pods/c8824800-8838-4200-8f50-4927e8075d3b/volumes"
Oct 30 06:20:04.145716 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:04.145672    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:04.297300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:04.297252    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:04.455154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:04.455049    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:04.601558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:04.601525    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:04.744237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:04.744134    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:05.073758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:05.073633    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:06.434566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.434503    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:20:06.532766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.532731    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-data\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.532966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.532795    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-certs\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.532966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.532839    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-envoy\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.532966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.532913    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-credential-socket\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.533169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.533039    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-istio-token\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-istio-token\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.533169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.533043    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-data" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:06.533169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.533081    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b50a167-d13a-4729-9583-92075bb0fd64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b50a167-d13a-4729-9583-92075bb0fd64-istiod-ca-cert\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.533169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.533136    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-kube-api-access-llnbc\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-kube-api-access-llnbc\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.533430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.533203    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b50a167-d13a-4729-9583-92075bb0fd64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b50a167-d13a-4729-9583-92075bb0fd64-istio-podinfo\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.533430 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.533233    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-socket\") pod \"4b50a167-d13a-4729-9583-92075bb0fd64\" (UID: \"4b50a167-d13a-4729-9583-92075bb0fd64\") "
Oct 30 06:20:06.534379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.534319    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.534379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.534331    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-credential-socket" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:06.534605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.534547    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-socket" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:06.534994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.534957    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b50a167-d13a-4729-9583-92075bb0fd64-istiod-ca-cert" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:20:06.535161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.535139    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-certs" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:06.539722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.539686    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-kube-api-access-llnbc" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "kube-api-access-llnbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:06.546480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.545760    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b50a167-d13a-4729-9583-92075bb0fd64-istio-podinfo" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:20:06.546480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.545827    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-istio-token" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:06.546480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.545901    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-envoy" pod "4b50a167-d13a-4729-9583-92075bb0fd64" (UID: "4b50a167-d13a-4729-9583-92075bb0fd64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:06.577816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.577781    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:06.636537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.636384    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.636537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.636421    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b50a167-d13a-4729-9583-92075bb0fd64-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.636537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.636435    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llnbc\" (UniqueName: \"kubernetes.io/projected/4b50a167-d13a-4729-9583-92075bb0fd64-kube-api-access-llnbc\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.636537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.636453    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b50a167-d13a-4729-9583-92075bb0fd64-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.636537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.636467    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.636537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.636481    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.636537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.636498    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.636537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.636510    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b50a167-d13a-4729-9583-92075bb0fd64-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:06.714670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.714513    5250 generic.go:356] "Generic (PLEG): container finished" podID="4b50a167-d13a-4729-9583-92075bb0fd64" containerID="d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1" exitCode=0
Oct 30 06:20:06.714670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.714575    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" event={"ID":"4b50a167-d13a-4729-9583-92075bb0fd64","Type":"ContainerDied","Data":"d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1"}
Oct 30 06:20:06.714670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.714628    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv" event={"ID":"4b50a167-d13a-4729-9583-92075bb0fd64","Type":"ContainerDied","Data":"f1858067a6f8ef231676a81922f9350183b3a5949690ba75eb8dcb69ad956e8a"}
Oct 30 06:20:06.714670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.714657    5250 scope.go:122] "RemoveContainer" containerID="d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1"
Oct 30 06:20:06.715083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.714790    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"
Oct 30 06:20:06.736729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.736702    5250 scope.go:122] "RemoveContainer" containerID="d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1"
Oct 30 06:20:06.738130 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:06.738096    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1\": container with ID starting with d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1 not found: ID does not exist" containerID="d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1"
Oct 30 06:20:06.738294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.738139    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1"} err="failed to get container status \"d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1\": rpc error: code = NotFound desc = could not find container \"d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1\": container with ID starting with d6153da603dbd5fb6c63c49142464126be401fbd6b481b2ba5afd41c4adf10f1 not found: ID does not exist"
Oct 30 06:20:06.767924 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.767872    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:06.773901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.773862    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"]
Oct 30 06:20:06.784553 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.784513    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-cqhbv"]
Oct 30 06:20:06.917811 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:06.917773    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:07.283461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:07.283415    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" path="/var/lib/kubelet/pods/4b50a167-d13a-4729-9583-92075bb0fd64/volumes"
Oct 30 06:20:07.284087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:07.284056    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:07.492354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:07.492302    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:07.493995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:07.493966    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"]
Oct 30 06:20:07.498026 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:07.497999    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36ae27-22gf4"]
Oct 30 06:20:07.637186 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:07.637150    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:07.780473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:07.780416    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:07.991166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:07.991052    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:09.282046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:09.282012    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d77e321-e8ce-49e5-98e0-34f236d8cfa1" path="/var/lib/kubelet/pods/4d77e321-e8ce-49e5-98e0-34f236d8cfa1/volumes"
Oct 30 06:20:09.746225 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:09.740915    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:09.889196 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:09.889155    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:10.031345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.031266    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.152911    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"]
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.153943    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.153966    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154009    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154018    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154028    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154036    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154066    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ea7617e8-53a7-4240-9cf3-03aceedc9153" containerName="revoke-keys"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154074    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7617e8-53a7-4240-9cf3-03aceedc9153" containerName="revoke-keys"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154267    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7617e8-53a7-4240-9cf3-03aceedc9153" containerName="revoke-keys"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154288    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b50a167-d13a-4729-9583-92075bb0fd64" containerName="istio-proxy"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154305    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c92033e-9264-4ae8-b0b6-779c547dba3c" containerName="main"
Oct 30 06:20:10.160204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.154317    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8824800-8838-4200-8f50-4927e8075d3b" containerName="main"
Oct 30 06:20:10.195815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.195784    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"]
Oct 30 06:20:10.196207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.196123    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:10.243200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.237641    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-718730\""
Oct 30 06:20:10.243504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.241502    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-718730-dockercfg-zcqw4\""
Oct 30 06:20:10.339012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.338909    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4e90f5-4af1-4377-b56e-fb0b01c41025-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7\" (UID: \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:10.339474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.339002    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xgt\" (UniqueName: \"kubernetes.io/projected/3d4e90f5-4af1-4377-b56e-fb0b01c41025-kube-api-access-66xgt\") pod \"payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7\" (UID: \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:10.440110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.440078    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4e90f5-4af1-4377-b56e-fb0b01c41025-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7\" (UID: \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:10.440332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.440147    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xgt\" (UniqueName: \"kubernetes.io/projected/3d4e90f5-4af1-4377-b56e-fb0b01c41025-kube-api-access-66xgt\") pod \"payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7\" (UID: \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:10.440829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.440811    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4e90f5-4af1-4377-b56e-fb0b01c41025-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7\" (UID: \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:10.461933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.461898    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xgt\" (UniqueName: \"kubernetes.io/projected/3d4e90f5-4af1-4377-b56e-fb0b01c41025-kube-api-access-66xgt\") pod \"payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7\" (UID: \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:10.544924 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.544893    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:10.849988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:10.849960    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"]
Oct 30 06:20:10.852709 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:20:10.852606    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4e90f5_4af1_4377_b56e_fb0b01c41025.slice/crio-8645ea036f1a81d3ab5e6a4b06b7c12153875c0a9b844875409d6f07dd3de61c WatchSource:0}: Error finding container 8645ea036f1a81d3ab5e6a4b06b7c12153875c0a9b844875409d6f07dd3de61c: Status 404 returned error can't find the container with id 8645ea036f1a81d3ab5e6a4b06b7c12153875c0a9b844875409d6f07dd3de61c
Oct 30 06:20:11.363475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:11.363440    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:11.772967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:11.772924    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7" event={"ID":"3d4e90f5-4af1-4377-b56e-fb0b01c41025","Type":"ContainerStarted","Data":"1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f"}
Oct 30 06:20:11.773156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:11.772987    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7" event={"ID":"3d4e90f5-4af1-4377-b56e-fb0b01c41025","Type":"ContainerStarted","Data":"8645ea036f1a81d3ab5e6a4b06b7c12153875c0a9b844875409d6f07dd3de61c"}
Oct 30 06:20:11.773156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:11.773012    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:11.810734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:11.810688    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7" podStartSLOduration=1.8106748719999999 podStartE2EDuration="1.810674872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:11.808777542 +0000 UTC m=+2926.639814760" watchObservedRunningTime="2026-10-30 06:20:11.810674872 +0000 UTC m=+2926.641712088"
Oct 30 06:20:12.106518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:12.106425    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"]
Oct 30 06:20:12.239383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:12.239345    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:12.386078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:12.386040    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:12.916863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:12.916830    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"]
Oct 30 06:20:13.012693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:13.012649    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:13.173103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:13.173026    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:14.905654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:14.905620    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:16.486584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:16.486546    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:17.106708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:17.106671    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft" podUID="8f39b589-6350-4092-b876-456fd1259aa5" containerName="payload-pre-processing" containerID="cri-o://1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c" gracePeriod=25
Oct 30 06:20:18.063892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:18.063859    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:19.915507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:19.915448    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:20.826441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:20.826403    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"]
Oct 30 06:20:20.826696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:20.826669    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy" containerID="cri-o://c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3" gracePeriod=30
Oct 30 06:20:21.018308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:21.018267    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft" podUID="8f39b589-6350-4092-b876-456fd1259aa5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:20:22.093942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:22.093912    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:22.785962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:22.785934    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:20:23.148897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:23.148862    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:23.303279 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:23.303225    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:24.209415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:24.209317    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:24.374941 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:24.374908    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:26.104780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.104711    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:20:26.195431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.195402    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80bd2dfe-d70f-4fd9-b75b-518282051732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bd2dfe-d70f-4fd9-b75b-518282051732-istiod-ca-cert\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.195616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.195491    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-token\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-token\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.195616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.195555    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-credential-socket\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.195742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.195618    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-podinfo\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.195742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.195703    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-certs\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.196219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.195846    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bd2dfe-d70f-4fd9-b75b-518282051732-istiod-ca-cert" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:20:26.196219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.195946    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-credential-socket" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:26.196219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.196104    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-certs" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:26.196642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.196232    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-kube-api-access-9vdnf\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-kube-api-access-9vdnf\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.196642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.196271    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-data\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.196642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.196294    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-envoy\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.196642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.196395    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-socket\") pod \"80bd2dfe-d70f-4fd9-b75b-518282051732\" (UID: \"80bd2dfe-d70f-4fd9-b75b-518282051732\") "
Oct 30 06:20:26.196868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.196749    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-data" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:26.197240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.197123    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-socket" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:26.197240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.197128    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.197240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.197169    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.197240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.197205    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.197240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.197218    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bd2dfe-d70f-4fd9-b75b-518282051732-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.198748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.198690    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-podinfo" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:20:26.198914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.198855    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-kube-api-access-9vdnf" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "kube-api-access-9vdnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:26.199163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.199141    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-token" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:26.199309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.199252    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-envoy" pod "80bd2dfe-d70f-4fd9-b75b-518282051732" (UID: "80bd2dfe-d70f-4fd9-b75b-518282051732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:26.298275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.298239    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vdnf\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-kube-api-access-9vdnf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.298275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.298269    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.298275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.298279    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bd2dfe-d70f-4fd9-b75b-518282051732-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.298532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.298287    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.298532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.298296    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bd2dfe-d70f-4fd9-b75b-518282051732-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:26.523424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.523385    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:26.898999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.898961    5250 generic.go:356] "Generic (PLEG): container finished" podID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerID="c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3" exitCode=0
Oct 30 06:20:26.899229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.899036    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"
Oct 30 06:20:26.899229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.899046    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" event={"ID":"80bd2dfe-d70f-4fd9-b75b-518282051732","Type":"ContainerDied","Data":"c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3"}
Oct 30 06:20:26.899229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.899104    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx" event={"ID":"80bd2dfe-d70f-4fd9-b75b-518282051732","Type":"ContainerDied","Data":"c46b935e862f18a3d37a4fd0e20df778bee6164dd4c07b34a0d5a8c3974fb632"}
Oct 30 06:20:26.899229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.899134    5250 scope.go:122] "RemoveContainer" containerID="c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3"
Oct 30 06:20:26.912450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.912430    5250 scope.go:122] "RemoveContainer" containerID="c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3"
Oct 30 06:20:26.912741 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:26.912725    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3\": container with ID starting with c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3 not found: ID does not exist" containerID="c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3"
Oct 30 06:20:26.912782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.912750    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3"} err="failed to get container status \"c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3\": rpc error: code = NotFound desc = could not find container \"c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3\": container with ID starting with c77a8d010e519cf4edb756586147cac363fe06985e88c4858cfe0720e8312bd3 not found: ID does not exist"
Oct 30 06:20:26.959949 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.959915    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"]
Oct 30 06:20:26.971072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:26.971039    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-c9jzx"]
Oct 30 06:20:27.282787 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:27.282695    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" path="/var/lib/kubelet/pods/80bd2dfe-d70f-4fd9-b75b-518282051732/volumes"
Oct 30 06:20:27.941707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:27.941664    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw" podUID="982646c7-61c8-40f6-851f-c599a18f7920" containerName="main" containerID="cri-o://d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28" gracePeriod=45
Oct 30 06:20:28.090285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.090148    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:28.112753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.112720    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw" podUID="982646c7-61c8-40f6-851f-c599a18f7920" containerName="main" probeResult="failure" output="Get \"https://10.133.1.5:8000/ready\": dial tcp 10.133.1.5:8000: connect: connection refused"
Oct 30 06:20:28.241379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.241354    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:20:28.328697 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.328661    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-tmp-dir\") pod \"982646c7-61c8-40f6-851f-c599a18f7920\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") "
Oct 30 06:20:28.329098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.328731    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-dshm\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-dshm\") pod \"982646c7-61c8-40f6-851f-c599a18f7920\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") "
Oct 30 06:20:28.329098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.328922    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-model-cache\") pod \"982646c7-61c8-40f6-851f-c599a18f7920\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") "
Oct 30 06:20:28.329098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.328982    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-tmp-dir" pod "982646c7-61c8-40f6-851f-c599a18f7920" (UID: "982646c7-61c8-40f6-851f-c599a18f7920"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:28.329098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.329034    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982646c7-61c8-40f6-851f-c599a18f7920-kube-api-access-f2xn9\" (UniqueName: \"kubernetes.io/projected/982646c7-61c8-40f6-851f-c599a18f7920-kube-api-access-f2xn9\") pod \"982646c7-61c8-40f6-851f-c599a18f7920\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") "
Oct 30 06:20:28.329352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.329220    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982646c7-61c8-40f6-851f-c599a18f7920-tls-certs\" (UniqueName: \"kubernetes.io/secret/982646c7-61c8-40f6-851f-c599a18f7920-tls-certs\") pod \"982646c7-61c8-40f6-851f-c599a18f7920\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") "
Oct 30 06:20:28.329352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.329255    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-home\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-home\") pod \"982646c7-61c8-40f6-851f-c599a18f7920\" (UID: \"982646c7-61c8-40f6-851f-c599a18f7920\") "
Oct 30 06:20:28.329969 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.329730    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-home" pod "982646c7-61c8-40f6-851f-c599a18f7920" (UID: "982646c7-61c8-40f6-851f-c599a18f7920"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:28.329969 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.329906    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:28.329969 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.329925    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:28.330389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.330359    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-model-cache" pod "982646c7-61c8-40f6-851f-c599a18f7920" (UID: "982646c7-61c8-40f6-851f-c599a18f7920"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:28.332266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.332165    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982646c7-61c8-40f6-851f-c599a18f7920-tls-certs" pod "982646c7-61c8-40f6-851f-c599a18f7920" (UID: "982646c7-61c8-40f6-851f-c599a18f7920"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:20:28.332749 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.332708    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-dshm" pod "982646c7-61c8-40f6-851f-c599a18f7920" (UID: "982646c7-61c8-40f6-851f-c599a18f7920"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:20:28.335335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.335313    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982646c7-61c8-40f6-851f-c599a18f7920-kube-api-access-f2xn9" pod "982646c7-61c8-40f6-851f-c599a18f7920" (UID: "982646c7-61c8-40f6-851f-c599a18f7920"). InnerVolumeSpecName "kube-api-access-f2xn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:28.431055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.431018    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:28.431055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.431050    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xn9\" (UniqueName: \"kubernetes.io/projected/982646c7-61c8-40f6-851f-c599a18f7920-kube-api-access-f2xn9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:28.431055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.431062    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982646c7-61c8-40f6-851f-c599a18f7920-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:28.431343 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.431071    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982646c7-61c8-40f6-851f-c599a18f7920-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:28.918616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.918579    5250 generic.go:356] "Generic (PLEG): container finished" podID="982646c7-61c8-40f6-851f-c599a18f7920" containerID="d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28" exitCode=0
Oct 30 06:20:28.918815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.918643    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"
Oct 30 06:20:28.918815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.918674    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw" event={"ID":"982646c7-61c8-40f6-851f-c599a18f7920","Type":"ContainerDied","Data":"d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28"}
Oct 30 06:20:28.918815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.918730    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw" event={"ID":"982646c7-61c8-40f6-851f-c599a18f7920","Type":"ContainerDied","Data":"f3bf1a97f0fe5edd836c33dfe01452580ccabc10ceb04652b36ed2efbe335f06"}
Oct 30 06:20:28.918815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.918755    5250 scope.go:122] "RemoveContainer" containerID="d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28"
Oct 30 06:20:28.932284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.932263    5250 scope.go:122] "RemoveContainer" containerID="d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28"
Oct 30 06:20:28.932606 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:28.932578    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28\": container with ID starting with d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28 not found: ID does not exist" containerID="d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28"
Oct 30 06:20:28.932714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.932619    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28"} err="failed to get container status \"d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28\": rpc error: code = NotFound desc = could not find container \"d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28\": container with ID starting with d7dd2d8b79bf5f18438af73b3b98a631c7e5554b2de695e530cb5b3d906dfb28 not found: ID does not exist"
Oct 30 06:20:28.967320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.967283    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"]
Oct 30 06:20:28.976928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:28.976900    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-36ae27/ipp-route-36ae27-kserve-7f74b87ff7-tzdgw"]
Oct 30 06:20:29.109061 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:29.109028    5250 scope.go:122] "RemoveContainer" containerID="7b32592f3d482c8dbd9435af55890b54ed67d9ca136207e871c89c323088a1bb"
Oct 30 06:20:29.119688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:29.119669    5250 scope.go:122] "RemoveContainer" containerID="bbe010621ceed714da992f50df024459e008e6155989f12af45ff101eb7d747a"
Oct 30 06:20:29.283436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:29.283399    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982646c7-61c8-40f6-851f-c599a18f7920" path="/var/lib/kubelet/pods/982646c7-61c8-40f6-851f-c599a18f7920/volumes"
Oct 30 06:20:29.797801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:29.797757    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:29.938328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:29.938301    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:31.018439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:31.018394    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft" podUID="8f39b589-6350-4092-b876-456fd1259aa5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:20:31.487979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:31.487942    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:32.374481 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:32.374444    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:20:33.486240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:33.486196    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:34.919913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:34.919875    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:35.547757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:35.547713    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:35.788709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:35.788679    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:35.936502 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:35.936473    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:36.311098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:36.311022    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:37.120586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.120556    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:37.382578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.382510    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 06:20:37.459582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.459555    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f39b589-6350-4092-b876-456fd1259aa5-kube-api-access-86txb\" (UniqueName: \"kubernetes.io/projected/8f39b589-6350-4092-b876-456fd1259aa5-kube-api-access-86txb\") pod \"8f39b589-6350-4092-b876-456fd1259aa5\" (UID: \"8f39b589-6350-4092-b876-456fd1259aa5\") "
Oct 30 06:20:37.459758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.459720    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f39b589-6350-4092-b876-456fd1259aa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f39b589-6350-4092-b876-456fd1259aa5-plugins-config-volume\") pod \"8f39b589-6350-4092-b876-456fd1259aa5\" (UID: \"8f39b589-6350-4092-b876-456fd1259aa5\") "
Oct 30 06:20:37.460215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.460161    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f39b589-6350-4092-b876-456fd1259aa5-plugins-config-volume" pod "8f39b589-6350-4092-b876-456fd1259aa5" (UID: "8f39b589-6350-4092-b876-456fd1259aa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:20:37.462100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.462069    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f39b589-6350-4092-b876-456fd1259aa5-kube-api-access-86txb" pod "8f39b589-6350-4092-b876-456fd1259aa5" (UID: "8f39b589-6350-4092-b876-456fd1259aa5"). InnerVolumeSpecName "kube-api-access-86txb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:37.561077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.561040    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f39b589-6350-4092-b876-456fd1259aa5-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:37.561077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.561073    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86txb\" (UniqueName: \"kubernetes.io/projected/8f39b589-6350-4092-b876-456fd1259aa5-kube-api-access-86txb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:37.617091 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.617058    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:37.757736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:37.757673    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:38.005116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.005081    5250 generic.go:356] "Generic (PLEG): container finished" podID="8f39b589-6350-4092-b876-456fd1259aa5" containerID="1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c" exitCode=0
Oct 30 06:20:38.005324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.005134    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"
Oct 30 06:20:38.005324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.005159    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft" event={"ID":"8f39b589-6350-4092-b876-456fd1259aa5","Type":"ContainerDied","Data":"1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c"}
Oct 30 06:20:38.005324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.005252    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft" event={"ID":"8f39b589-6350-4092-b876-456fd1259aa5","Type":"ContainerDied","Data":"46c0443af453b3ecc360461c5c0f4781caa0a3a2568a5c711096eb1e0efaf2ca"}
Oct 30 06:20:38.005324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.005276    5250 scope.go:122] "RemoveContainer" containerID="1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c"
Oct 30 06:20:38.016756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.016737    5250 scope.go:122] "RemoveContainer" containerID="1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c"
Oct 30 06:20:38.017025 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:38.017003    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c\": container with ID starting with 1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c not found: ID does not exist" containerID="1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c"
Oct 30 06:20:38.017107 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.017031    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c"} err="failed to get container status \"1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c\": rpc error: code = NotFound desc = could not find container \"1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c\": container with ID starting with 1931464caf16b43a7135005fb296f264360d6da0c903b64a10237b4c47fd3e2c not found: ID does not exist"
Oct 30 06:20:38.053257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.053231    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"]
Oct 30 06:20:38.063110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:38.063085    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-1b4b7a-8499f6d75-vczft"]
Oct 30 06:20:39.105737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:39.105703    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:39.282169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:39.282130    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f39b589-6350-4092-b876-456fd1259aa5" path="/var/lib/kubelet/pods/8f39b589-6350-4092-b876-456fd1259aa5/volumes"
Oct 30 06:20:40.869712 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:40.869675    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:42.585526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:42.585487    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:42.719092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:42.719050    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:42.938295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:42.938246    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:43.899425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:43.899390    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:44.085484 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:44.059189    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:44.931804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:44.931773    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:45.078202 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:45.074291    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:45.218930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:45.218842    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:46.580070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:46.580032    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:47.992826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:47.992786    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:48.647590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:48.647544    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:48.948858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:48.948778    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:49.095104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:49.095075    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:49.458745 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:49.458710    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:50.176816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:50.176784    5250 generic.go:356] "Generic (PLEG): container finished" podID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerID="19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565" exitCode=0
Oct 30 06:20:50.177350 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:50.176864    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" event={"ID":"674faa9d-8c41-44bd-8d47-07c8c671fdc5","Type":"ContainerDied","Data":"19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565"}
Oct 30 06:20:50.177410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:50.177388    5250 scope.go:122] "RemoveContainer" containerID="19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565"
Oct 30 06:20:50.179374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:50.179342    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:20:50.179496 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:50.179433    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:20:50.179569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:50.179510    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:20:50.760197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:50.760146    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:50.974523 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:50.974469    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:51.189663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:51.189620    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" event={"ID":"674faa9d-8c41-44bd-8d47-07c8c671fdc5","Type":"ContainerStarted","Data":"efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750"}
Oct 30 06:20:51.815445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:51.815412    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:20:51.816353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:51.816317    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 06:20:51.816485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:51.816417    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 06:20:52.771201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:52.765992    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:52.814896 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:52.814862    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 06:20:52.815082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:52.814927    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 06:20:53.517768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:53.517729    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"]
Oct 30 06:20:53.544913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:53.544879    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"]
Oct 30 06:20:53.687731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:53.687657    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:53.814989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:53.814898    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 06:20:53.814989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:53.814961    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 06:20:53.835367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:53.835331    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:53.986819 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:53.986787    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:54.493229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:54.488351    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:54.814954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:54.814867    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 06:20:54.814954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:54.814938    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 06:20:55.814998 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:55.814962    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 06:20:55.815313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:55.815017    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 06:20:56.181581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:56.181530    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:56.280206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:56.278282    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:20:56.341036 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:56.340998    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:56.737204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:56.726912    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:57.121324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:57.121248    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:20:57.263051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:57.263021    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:20:57.694542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:57.694510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:57.833465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:57.833430    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:57.945088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:57.944168    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"]
Oct 30 06:20:57.945088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:57.944456    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" containerID="cri-o://8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c" gracePeriod=30
Oct 30 06:20:58.010097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.010061    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:58.035097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.035065    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"]
Oct 30 06:20:58.038473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.038446    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2869fa-q9m98"]
Oct 30 06:20:58.177670 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.177634    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:20:58.519032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.518993    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq" podUID="6763a628-af89-489c-afd2-a60dfb5370da" containerName="payload-processing" containerID="cri-o://fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b" gracePeriod=25
Oct 30 06:20:58.545544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.545514    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28" podUID="00655df9-4ba7-4c1b-b23b-b48315f0129d" containerName="payload-pre-processing" containerID="cri-o://11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974" gracePeriod=25
Oct 30 06:20:58.782105 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.782005    5250 patch_prober.go:29] interesting pod/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:20:58.782989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.782682    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:20:58.836155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.836130    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:20:58.872422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.872398    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:20:58.901394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.901362    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00655df9-4ba7-4c1b-b23b-b48315f0129d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00655df9-4ba7-4c1b-b23b-b48315f0129d-plugins-config-volume\") pod \"00655df9-4ba7-4c1b-b23b-b48315f0129d\" (UID: \"00655df9-4ba7-4c1b-b23b-b48315f0129d\") "
Oct 30 06:20:58.901594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.901450    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6763a628-af89-489c-afd2-a60dfb5370da-kube-api-access-crbm4\" (UniqueName: \"kubernetes.io/projected/6763a628-af89-489c-afd2-a60dfb5370da-kube-api-access-crbm4\") pod \"6763a628-af89-489c-afd2-a60dfb5370da\" (UID: \"6763a628-af89-489c-afd2-a60dfb5370da\") "
Oct 30 06:20:58.901594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.901520    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6763a628-af89-489c-afd2-a60dfb5370da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763a628-af89-489c-afd2-a60dfb5370da-plugins-config-volume\") pod \"6763a628-af89-489c-afd2-a60dfb5370da\" (UID: \"6763a628-af89-489c-afd2-a60dfb5370da\") "
Oct 30 06:20:58.901594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.901548    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00655df9-4ba7-4c1b-b23b-b48315f0129d-kube-api-access-w9mq6\" (UniqueName: \"kubernetes.io/projected/00655df9-4ba7-4c1b-b23b-b48315f0129d-kube-api-access-w9mq6\") pod \"00655df9-4ba7-4c1b-b23b-b48315f0129d\" (UID: \"00655df9-4ba7-4c1b-b23b-b48315f0129d\") "
Oct 30 06:20:58.901796 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.901763    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00655df9-4ba7-4c1b-b23b-b48315f0129d-plugins-config-volume" pod "00655df9-4ba7-4c1b-b23b-b48315f0129d" (UID: "00655df9-4ba7-4c1b-b23b-b48315f0129d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:20:58.902156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.902127    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6763a628-af89-489c-afd2-a60dfb5370da-plugins-config-volume" pod "6763a628-af89-489c-afd2-a60dfb5370da" (UID: "6763a628-af89-489c-afd2-a60dfb5370da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:20:58.902499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.902475    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6763a628-af89-489c-afd2-a60dfb5370da-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:58.902603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.902505    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00655df9-4ba7-4c1b-b23b-b48315f0129d-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:58.904872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.904851    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6763a628-af89-489c-afd2-a60dfb5370da-kube-api-access-crbm4" pod "6763a628-af89-489c-afd2-a60dfb5370da" (UID: "6763a628-af89-489c-afd2-a60dfb5370da"). InnerVolumeSpecName "kube-api-access-crbm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:58.904971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:58.904894    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00655df9-4ba7-4c1b-b23b-b48315f0129d-kube-api-access-w9mq6" pod "00655df9-4ba7-4c1b-b23b-b48315f0129d" (UID: "00655df9-4ba7-4c1b-b23b-b48315f0129d"). InnerVolumeSpecName "kube-api-access-w9mq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:20:59.003581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.003548    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crbm4\" (UniqueName: \"kubernetes.io/projected/6763a628-af89-489c-afd2-a60dfb5370da-kube-api-access-crbm4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:59.003581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.003579    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9mq6\" (UniqueName: \"kubernetes.io/projected/00655df9-4ba7-4c1b-b23b-b48315f0129d-kube-api-access-w9mq6\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:20:59.281077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.281041    5250 generic.go:356] "Generic (PLEG): container finished" podID="00655df9-4ba7-4c1b-b23b-b48315f0129d" containerID="11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974" exitCode=0
Oct 30 06:20:59.281563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.281139    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"
Oct 30 06:20:59.282982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.282960    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2" path="/var/lib/kubelet/pods/9f1651bb-7e96-4bd1-9ac4-39a0b5d0d1b2/volumes"
Oct 30 06:20:59.282982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.282967    5250 generic.go:356] "Generic (PLEG): container finished" podID="6763a628-af89-489c-afd2-a60dfb5370da" containerID="fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b" exitCode=0
Oct 30 06:20:59.283167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.283065    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"
Oct 30 06:20:59.283638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.283615    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28" event={"ID":"00655df9-4ba7-4c1b-b23b-b48315f0129d","Type":"ContainerDied","Data":"11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974"}
Oct 30 06:20:59.283708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.283665    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28" event={"ID":"00655df9-4ba7-4c1b-b23b-b48315f0129d","Type":"ContainerDied","Data":"435a2880b938fcace574a415a2462eb2decbf8dbcc7e69c3b1021786a64776e7"}
Oct 30 06:20:59.283708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.283691    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq" event={"ID":"6763a628-af89-489c-afd2-a60dfb5370da","Type":"ContainerDied","Data":"fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b"}
Oct 30 06:20:59.283801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.283717    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq" event={"ID":"6763a628-af89-489c-afd2-a60dfb5370da","Type":"ContainerDied","Data":"da1af2d318764b839b87ae8c8986cc0b44f1643e9aec238f84c179cd6de4394f"}
Oct 30 06:20:59.283801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.283742    5250 scope.go:122] "RemoveContainer" containerID="11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974"
Oct 30 06:20:59.297050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.297030    5250 scope.go:122] "RemoveContainer" containerID="11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974"
Oct 30 06:20:59.297408 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:59.297378    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974\": container with ID starting with 11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974 not found: ID does not exist" containerID="11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974"
Oct 30 06:20:59.297509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.297418    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974"} err="failed to get container status \"11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974\": rpc error: code = NotFound desc = could not find container \"11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974\": container with ID starting with 11c470847010746dc55c402debb0f34215e40511d8e02156ed4a42ef4b81f974 not found: ID does not exist"
Oct 30 06:20:59.297509 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.297442    5250 scope.go:122] "RemoveContainer" containerID="fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b"
Oct 30 06:20:59.307577 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.307557    5250 scope.go:122] "RemoveContainer" containerID="fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b"
Oct 30 06:20:59.307869 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:20:59.307848    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b\": container with ID starting with fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b not found: ID does not exist" containerID="fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b"
Oct 30 06:20:59.307947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.307880    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b"} err="failed to get container status \"fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b\": rpc error: code = NotFound desc = could not find container \"fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b\": container with ID starting with fe9ae3de4a73a85333390b2d18bdaf7d405682837adc964255f6b6e15d41881b not found: ID does not exist"
Oct 30 06:20:59.339524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.339501    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"]
Oct 30 06:20:59.349301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.349281    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-7fbcf5f66d-r64vq"]
Oct 30 06:20:59.379451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.379430    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"]
Oct 30 06:20:59.382837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.382818    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-84cd85d6b7-tkq28"]
Oct 30 06:20:59.790557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:20:59.790455    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:01.282834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:01.282803    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00655df9-4ba7-4c1b-b23b-b48315f0129d" path="/var/lib/kubelet/pods/00655df9-4ba7-4c1b-b23b-b48315f0129d/volumes"
Oct 30 06:21:01.283267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:01.283252    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6763a628-af89-489c-afd2-a60dfb5370da" path="/var/lib/kubelet/pods/6763a628-af89-489c-afd2-a60dfb5370da/volumes"
Oct 30 06:21:01.554547 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:01.554464    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:02.374823 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:02.374782    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:21:03.232464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.232400    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:21:03.265097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265066    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-podinfo\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265163    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-kube-api-access-h6gtg\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-kube-api-access-h6gtg\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265380    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-envoy\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265417    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-socket\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265492    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-data\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265528    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966f242f-1a7a-4a18-93ae-66d06f77f88f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f242f-1a7a-4a18-93ae-66d06f77f88f-istiod-ca-cert\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265604    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-token\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-token\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265633    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-credential-socket\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265724    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-certs\") pod \"966f242f-1a7a-4a18-93ae-66d06f77f88f\" (UID: \"966f242f-1a7a-4a18-93ae-66d06f77f88f\") "
Oct 30 06:21:03.265765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265737    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-data" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:03.265914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.265793    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-socket" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:03.266157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.266133    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-certs" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:03.266286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.266219    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-credential-socket" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:03.266505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.266480    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.266556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.266512    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.266556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.266526    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.266556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.266541    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.266673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.266546    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966f242f-1a7a-4a18-93ae-66d06f77f88f-istiod-ca-cert" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:21:03.267644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.267623    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-podinfo" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:21:03.267721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.267669    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-envoy" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:03.268029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.268010    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-kube-api-access-h6gtg" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "kube-api-access-h6gtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:03.268482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.268460    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-token" pod "966f242f-1a7a-4a18-93ae-66d06f77f88f" (UID: "966f242f-1a7a-4a18-93ae-66d06f77f88f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:03.320997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.320624    5250 generic.go:356] "Generic (PLEG): container finished" podID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerID="8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c" exitCode=0
Oct 30 06:21:03.320997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.320691    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"
Oct 30 06:21:03.320997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.320713    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" event={"ID":"966f242f-1a7a-4a18-93ae-66d06f77f88f","Type":"ContainerDied","Data":"8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c"}
Oct 30 06:21:03.320997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.320796    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7" event={"ID":"966f242f-1a7a-4a18-93ae-66d06f77f88f","Type":"ContainerDied","Data":"ff73f47485937d23daf5cd44291f2587e83de79135cc9aff9e116f33ed6ac4c3"}
Oct 30 06:21:03.320997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.320829    5250 scope.go:122] "RemoveContainer" containerID="8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c"
Oct 30 06:21:03.339884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.339860    5250 scope.go:122] "RemoveContainer" containerID="f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b"
Oct 30 06:21:03.350403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.350381    5250 scope.go:122] "RemoveContainer" containerID="8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c"
Oct 30 06:21:03.350660 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:03.350641    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c\": container with ID starting with 8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c not found: ID does not exist" containerID="8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c"
Oct 30 06:21:03.350714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.350671    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c"} err="failed to get container status \"8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c\": rpc error: code = NotFound desc = could not find container \"8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c\": container with ID starting with 8101c327fece544fc0cfca578c3c7cd7dd2a35c950ccf38bf59c011baadb525c not found: ID does not exist"
Oct 30 06:21:03.350714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.350695    5250 scope.go:122] "RemoveContainer" containerID="f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b"
Oct 30 06:21:03.350918 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:03.350903    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b\": container with ID starting with f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b not found: ID does not exist" containerID="f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b"
Oct 30 06:21:03.350961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.350922    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b"} err="failed to get container status \"f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b\": rpc error: code = NotFound desc = could not find container \"f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b\": container with ID starting with f485485cb60fa5702338af065c9d4c3293b56ad7097c78e7e6b72056411ca83b not found: ID does not exist"
Oct 30 06:21:03.367921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.367895    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.367921 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.367924    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.368391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.367939    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6gtg\" (UniqueName: \"kubernetes.io/projected/966f242f-1a7a-4a18-93ae-66d06f77f88f-kube-api-access-h6gtg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.368391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.367953    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f242f-1a7a-4a18-93ae-66d06f77f88f-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.368391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.367966    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f242f-1a7a-4a18-93ae-66d06f77f88f-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:03.372089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.372068    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"]
Oct 30 06:21:03.375835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.375816    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-36ae27-openshift-default-6c9566b779-w9hc7"]
Oct 30 06:21:03.816742 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:03.816691    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:05.032067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:05.032032    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"]
Oct 30 06:21:05.041932 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:05.041908    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a9dac2-5skwd"]
Oct 30 06:21:05.281959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:05.281930    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" path="/var/lib/kubelet/pods/966f242f-1a7a-4a18-93ae-66d06f77f88f/volumes"
Oct 30 06:21:05.282459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:05.282417    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8223c9b-9911-4274-b6a7-cb0161194e66" path="/var/lib/kubelet/pods/e8223c9b-9911-4274-b6a7-cb0161194e66/volumes"
Oct 30 06:21:05.960632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:05.960538    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:07.619383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:07.619341    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:07.759132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:07.759083    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:08.572928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:08.572897    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:10.377152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:10.377104    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:11.870511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:11.870460    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:13.587384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:13.587346    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:14.022172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:14.022135    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:16.008955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:16.008908    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:17.631371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:17.631331    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:19.268643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:19.268607    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:20.968480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:20.968439    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:22.562831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:22.562798    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:24.341347 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:24.341309    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:25.976607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:25.976582    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:21:26.180116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:26.180093    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:21:26.813302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:26.813247    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:28.381156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:28.381119    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:29.194239 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:29.194200    5250 scope.go:122] "RemoveContainer" containerID="0fd89ee66aa98c3e8411b8d9f33f0586ab035f678a75bc1b825b87d7bf6709aa"
Oct 30 06:21:29.205427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:29.205407    5250 scope.go:122] "RemoveContainer" containerID="49c2de2376e2ef78f5cb9cbb0591f128115e3fd92dd3807c608018843f2c5356"
Oct 30 06:21:29.215348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:29.215250    5250 scope.go:122] "RemoveContainer" containerID="0ee76eeec75ca38d1e60dcde4c627dd895911be1ad4c9f3ae16247b126471243"
Oct 30 06:21:29.225671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:29.225645    5250 scope.go:122] "RemoveContainer" containerID="3ed950be43e5538e064d7683ce0af00163d7a12e26c6c45be862cd3d418c4523"
Oct 30 06:21:29.235361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:29.235342    5250 scope.go:122] "RemoveContainer" containerID="759dbcc523bd0c0335b31eb6fa0e74e5a7fb78b357bed857ade29c9a211a766d"
Oct 30 06:21:30.171835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:30.171807    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:30.757237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:30.757201    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"]
Oct 30 06:21:30.916672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:30.916642    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:31.060754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:31.060671    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:31.206546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:31.206513    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:31.379907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:31.379824    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:31.588658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:31.588612    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:32.373956 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:32.373915    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:21:32.594526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:32.594496    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:34.203741 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:34.203707    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:35.034798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:35.034759    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"]
Oct 30 06:21:35.044039 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:35.044011    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfd81e0-47t75"]
Oct 30 06:21:35.281137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:35.281103    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b2e5df-11ba-4d51-8201-041991e29895" path="/var/lib/kubelet/pods/59b2e5df-11ba-4d51-8201-041991e29895/volumes"
Oct 30 06:21:35.561722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:35.561683    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:35.758286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:35.758238    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7" podUID="3d4e90f5-4af1-4377-b56e-fb0b01c41025" containerName="payload-processing" containerID="cri-o://1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f" gracePeriod=25
Oct 30 06:21:36.095672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.095649    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:21:36.154766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.154734    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:36.174099 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.174063    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4e90f5-4af1-4377-b56e-fb0b01c41025-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4e90f5-4af1-4377-b56e-fb0b01c41025-plugins-config-volume\") pod \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\" (UID: \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\") "
Oct 30 06:21:36.174320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.174197    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4e90f5-4af1-4377-b56e-fb0b01c41025-kube-api-access-66xgt\" (UniqueName: \"kubernetes.io/projected/3d4e90f5-4af1-4377-b56e-fb0b01c41025-kube-api-access-66xgt\") pod \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\" (UID: \"3d4e90f5-4af1-4377-b56e-fb0b01c41025\") "
Oct 30 06:21:36.174597 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.174572    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4e90f5-4af1-4377-b56e-fb0b01c41025-plugins-config-volume" pod "3d4e90f5-4af1-4377-b56e-fb0b01c41025" (UID: "3d4e90f5-4af1-4377-b56e-fb0b01c41025"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:21:36.175044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.175005    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4e90f5-4af1-4377-b56e-fb0b01c41025-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:36.176638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.176613    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4e90f5-4af1-4377-b56e-fb0b01c41025-kube-api-access-66xgt" pod "3d4e90f5-4af1-4377-b56e-fb0b01c41025" (UID: "3d4e90f5-4af1-4377-b56e-fb0b01c41025"). InnerVolumeSpecName "kube-api-access-66xgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:36.279050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.276907    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66xgt\" (UniqueName: \"kubernetes.io/projected/3d4e90f5-4af1-4377-b56e-fb0b01c41025-kube-api-access-66xgt\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:36.395393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.395356    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:36.629720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.629628    5250 generic.go:356] "Generic (PLEG): container finished" podID="3d4e90f5-4af1-4377-b56e-fb0b01c41025" containerID="1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f" exitCode=0
Oct 30 06:21:36.629720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.629696    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"
Oct 30 06:21:36.629933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.629718    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7" event={"ID":"3d4e90f5-4af1-4377-b56e-fb0b01c41025","Type":"ContainerDied","Data":"1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f"}
Oct 30 06:21:36.629933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.629798    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7" event={"ID":"3d4e90f5-4af1-4377-b56e-fb0b01c41025","Type":"ContainerDied","Data":"8645ea036f1a81d3ab5e6a4b06b7c12153875c0a9b844875409d6f07dd3de61c"}
Oct 30 06:21:36.629933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.629821    5250 scope.go:122] "RemoveContainer" containerID="1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f"
Oct 30 06:21:36.640641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.640620    5250 scope.go:122] "RemoveContainer" containerID="1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f"
Oct 30 06:21:36.640911 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:36.640894    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f\": container with ID starting with 1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f not found: ID does not exist" containerID="1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f"
Oct 30 06:21:36.640958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.640920    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f"} err="failed to get container status \"1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f\": rpc error: code = NotFound desc = could not find container \"1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f\": container with ID starting with 1f78e9bd651cea235c81bda15f7de50dbe171c99cb15c3e26c2b092561209e2f not found: ID does not exist"
Oct 30 06:21:36.676403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.676380    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"]
Oct 30 06:21:36.680713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.680693    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:36.681306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.681289    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-ff64f4586-pbhw7"]
Oct 30 06:21:36.936357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:36.936320    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:37.237515 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:37.237436    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:37.282014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:37.281985    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4e90f5-4af1-4377-b56e-fb0b01c41025" path="/var/lib/kubelet/pods/3d4e90f5-4af1-4377-b56e-fb0b01c41025/volumes"
Oct 30 06:21:38.030266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:38.030234    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"]
Oct 30 06:21:38.039809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:38.039778    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b0bda6-6q2xc"]
Oct 30 06:21:38.267361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:38.267326    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:39.281889 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:39.281856    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d10798a-4aba-4df7-93d6-3763fb8df4f0" path="/var/lib/kubelet/pods/8d10798a-4aba-4df7-93d6-3763fb8df4f0/volumes"
Oct 30 06:21:39.738577 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:39.738539    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:41.006015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.005985    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"]
Oct 30 06:21:41.006490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.006268    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6" podUID="0ed3595f-719d-4076-abc8-f4ee6913c5ad" containerName="maas-api" containerID="cri-o://b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0" gracePeriod=30
Oct 30 06:21:41.099519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.099486    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:41.277459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.277332    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:41.337442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.337417    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:21:41.348680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.348653    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed3595f-719d-4076-abc8-f4ee6913c5ad-kube-api-access-m4d2j\" (UniqueName: \"kubernetes.io/projected/0ed3595f-719d-4076-abc8-f4ee6913c5ad-kube-api-access-m4d2j\") pod \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") "
Oct 30 06:21:41.348815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.348693    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-maas-api-tls\") pod \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") "
Oct 30 06:21:41.348815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.348723    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed3595f-719d-4076-abc8-f4ee6913c5ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed3595f-719d-4076-abc8-f4ee6913c5ad-openshift-service-ca\") pod \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") "
Oct 30 06:21:41.348815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.348782    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-metrics-certs\") pod \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\" (UID: \"0ed3595f-719d-4076-abc8-f4ee6913c5ad\") "
Oct 30 06:21:41.349152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.349124    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed3595f-719d-4076-abc8-f4ee6913c5ad-openshift-service-ca" pod "0ed3595f-719d-4076-abc8-f4ee6913c5ad" (UID: "0ed3595f-719d-4076-abc8-f4ee6913c5ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:21:41.349437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.349365    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed3595f-719d-4076-abc8-f4ee6913c5ad-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:41.351096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.351066    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed3595f-719d-4076-abc8-f4ee6913c5ad-kube-api-access-m4d2j" pod "0ed3595f-719d-4076-abc8-f4ee6913c5ad" (UID: "0ed3595f-719d-4076-abc8-f4ee6913c5ad"). InnerVolumeSpecName "kube-api-access-m4d2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:41.351539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.351512    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-metrics-certs" pod "0ed3595f-719d-4076-abc8-f4ee6913c5ad" (UID: "0ed3595f-719d-4076-abc8-f4ee6913c5ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:21:41.351630 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.351539    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-maas-api-tls" pod "0ed3595f-719d-4076-abc8-f4ee6913c5ad" (UID: "0ed3595f-719d-4076-abc8-f4ee6913c5ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:21:41.417877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.417835    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:41.450234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.450163    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4d2j\" (UniqueName: \"kubernetes.io/projected/0ed3595f-719d-4076-abc8-f4ee6913c5ad-kube-api-access-m4d2j\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:41.450234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.450224    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:41.450234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.450241    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed3595f-719d-4076-abc8-f4ee6913c5ad-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:41.603290 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.603167    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:41.716440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.716408    5250 generic.go:356] "Generic (PLEG): container finished" podID="0ed3595f-719d-4076-abc8-f4ee6913c5ad" containerID="b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0" exitCode=0
Oct 30 06:21:41.716610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.716462    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"
Oct 30 06:21:41.716610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.716465    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6" event={"ID":"0ed3595f-719d-4076-abc8-f4ee6913c5ad","Type":"ContainerDied","Data":"b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0"}
Oct 30 06:21:41.716610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.716518    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6" event={"ID":"0ed3595f-719d-4076-abc8-f4ee6913c5ad","Type":"ContainerDied","Data":"6dd968faeae0622192eb9053e236bf068794596a6a6857b5f3eaad0cf2bbb67a"}
Oct 30 06:21:41.716610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.716547    5250 scope.go:122] "RemoveContainer" containerID="b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0"
Oct 30 06:21:41.727923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.727904    5250 scope.go:122] "RemoveContainer" containerID="b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0"
Oct 30 06:21:41.728197 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:41.728163    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0\": container with ID starting with b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0 not found: ID does not exist" containerID="b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0"
Oct 30 06:21:41.728261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.728207    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0"} err="failed to get container status \"b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0\": rpc error: code = NotFound desc = could not find container \"b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0\": container with ID starting with b4145506461dda6ee5cb91ad90c52151715cfb0b48512c28c7c86b327d473cb0 not found: ID does not exist"
Oct 30 06:21:41.757776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.757751    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:41.763337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.763312    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"]
Oct 30 06:21:41.769463 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.769439    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5a591d-6d786b8979-m7vx6"]
Oct 30 06:21:41.900296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:41.900263    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:42.303251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:42.303155    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:43.281932 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:43.281897    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed3595f-719d-4076-abc8-f4ee6913c5ad" path="/var/lib/kubelet/pods/0ed3595f-719d-4076-abc8-f4ee6913c5ad/volumes"
Oct 30 06:21:43.663076 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:43.663022    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:43.802247 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:43.802207    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:43.942693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:43.942604    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:43.950269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:43.947260    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"]
Oct 30 06:21:43.950696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:43.950663    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy" containerID="cri-o://f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e" gracePeriod=30
Oct 30 06:21:44.094487 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:44.094459    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:44.429373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:44.429341    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:44.570582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:44.570540    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:44.711778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:44.711694    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:45.057334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:45.057246    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:46.291841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.291804    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:46.498297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498256    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"]
Oct 30 06:21:46.498909 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498894    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8f39b589-6350-4092-b876-456fd1259aa5" containerName="payload-pre-processing"
Oct 30 06:21:46.498952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498911    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f39b589-6350-4092-b876-456fd1259aa5" containerName="payload-pre-processing"
Oct 30 06:21:46.498952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498926    5250 cpu_manager.go:414] "containerMap: removing container" podUID="00655df9-4ba7-4c1b-b23b-b48315f0129d" containerName="payload-pre-processing"
Oct 30 06:21:46.498952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498931    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00655df9-4ba7-4c1b-b23b-b48315f0129d" containerName="payload-pre-processing"
Oct 30 06:21:46.498952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498942    5250 cpu_manager.go:414] "containerMap: removing container" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy"
Oct 30 06:21:46.498952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498947    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498956    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6763a628-af89-489c-afd2-a60dfb5370da" containerName="payload-processing"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498961    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6763a628-af89-489c-afd2-a60dfb5370da" containerName="payload-processing"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498969    5250 cpu_manager.go:414] "containerMap: removing container" podUID="982646c7-61c8-40f6-851f-c599a18f7920" containerName="main"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498973    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982646c7-61c8-40f6-851f-c599a18f7920" containerName="main"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498979    5250 cpu_manager.go:414] "containerMap: removing container" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.498983    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499008    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3d4e90f5-4af1-4377-b56e-fb0b01c41025" containerName="payload-processing"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499012    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4e90f5-4af1-4377-b56e-fb0b01c41025" containerName="payload-processing"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499023    5250 cpu_manager.go:414] "containerMap: removing container" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499028    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499036    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0ed3595f-719d-4076-abc8-f4ee6913c5ad" containerName="maas-api"
Oct 30 06:21:46.499134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499040    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed3595f-719d-4076-abc8-f4ee6913c5ad" containerName="maas-api"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499138    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f39b589-6350-4092-b876-456fd1259aa5" containerName="payload-pre-processing"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499150    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="00655df9-4ba7-4c1b-b23b-b48315f0129d" containerName="payload-pre-processing"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499157    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499163    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6763a628-af89-489c-afd2-a60dfb5370da" containerName="payload-processing"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499170    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="982646c7-61c8-40f6-851f-c599a18f7920" containerName="main"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499189    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="966f242f-1a7a-4a18-93ae-66d06f77f88f" containerName="istio-proxy"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499198    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4e90f5-4af1-4377-b56e-fb0b01c41025" containerName="payload-processing"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499204    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="80bd2dfe-d70f-4fd9-b75b-518282051732" containerName="istio-proxy"
Oct 30 06:21:46.499556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.499212    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed3595f-719d-4076-abc8-f4ee6913c5ad" containerName="maas-api"
Oct 30 06:21:46.504130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.504112    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.520934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.520906    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"]
Oct 30 06:21:46.525958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.525930    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:21:46.525958 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.525940    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:21:46.542616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.542536    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.542616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.542576    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.542794 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.542709    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zvz\" (UniqueName: \"kubernetes.io/projected/b29e4618-4c76-4223-ad25-0a24881d1eb4-kube-api-access-v4zvz\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.644556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.644515    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.644759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.644568    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.644759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.644686    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zvz\" (UniqueName: \"kubernetes.io/projected/b29e4618-4c76-4223-ad25-0a24881d1eb4-kube-api-access-v4zvz\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.645444 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.645395    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.647067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.647044    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.665911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.665881    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zvz\" (UniqueName: \"kubernetes.io/projected/b29e4618-4c76-4223-ad25-0a24881d1eb4-kube-api-access-v4zvz\") pod \"maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:46.820723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:46.820633    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:47.133442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:47.133409    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"]
Oct 30 06:21:47.135884 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:21:47.135853    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29e4618_4c76_4223_ad25_0a24881d1eb4.slice/crio-5c9f6e4bf76f5c0b2a40d19213739908ceced97016eda8e71b17062b0e2fb513 WatchSource:0}: Error finding container 5c9f6e4bf76f5c0b2a40d19213739908ceced97016eda8e71b17062b0e2fb513: Status 404 returned error can't find the container with id 5c9f6e4bf76f5c0b2a40d19213739908ceced97016eda8e71b17062b0e2fb513
Oct 30 06:21:47.770302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:47.770212    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r" event={"ID":"b29e4618-4c76-4223-ad25-0a24881d1eb4","Type":"ContainerStarted","Data":"13e0a8b484d8727d6332407b0e49e051629ae7974d5ab9567cc67fc77e3fb56b"}
Oct 30 06:21:47.770302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:47.770276    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r" event={"ID":"b29e4618-4c76-4223-ad25-0a24881d1eb4","Type":"ContainerStarted","Data":"5c9f6e4bf76f5c0b2a40d19213739908ceced97016eda8e71b17062b0e2fb513"}
Oct 30 06:21:47.809066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:47.809018    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r" podStartSLOduration=1.8090038069999999 podStartE2EDuration="1.809003807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:47.805711997 +0000 UTC m=+3022.636749214" watchObservedRunningTime="2026-10-30 06:21:47.809003807 +0000 UTC m=+3022.640041024"
Oct 30 06:21:48.091508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:48.091476    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:48.740687 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:48.740654    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:48.787550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:48.787513    5250 generic.go:356] "Generic (PLEG): container finished" podID="b29e4618-4c76-4223-ad25-0a24881d1eb4" containerID="13e0a8b484d8727d6332407b0e49e051629ae7974d5ab9567cc67fc77e3fb56b" exitCode=0
Oct 30 06:21:48.788044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:48.787741    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r" event={"ID":"b29e4618-4c76-4223-ad25-0a24881d1eb4","Type":"ContainerDied","Data":"13e0a8b484d8727d6332407b0e49e051629ae7974d5ab9567cc67fc77e3fb56b"}
Oct 30 06:21:48.919601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:48.919567    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:48.989480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:48.989452    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"]
Oct 30 06:21:48.989975 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:48.989941    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy" containerID="cri-o://5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7" gracePeriod=30
Oct 30 06:21:49.061656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.061620    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:49.330723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.330698    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:21:49.410968 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.410936    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-certs\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411006    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-kube-api-access-zqn4x\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-kube-api-access-zqn4x\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411108    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-data\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411170    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-socket\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411251    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-podinfo\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411295    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-certs" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:49.411334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411302    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-credential-socket\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411390    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-token\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-token\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411485    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-credential-socket" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:49.411554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411487    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-envoy\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411616    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e377ba94-aefe-4012-b1eb-363b2ba740d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377ba94-aefe-4012-b1eb-363b2ba740d8-istiod-ca-cert\") pod \"e377ba94-aefe-4012-b1eb-363b2ba740d8\" (UID: \"e377ba94-aefe-4012-b1eb-363b2ba740d8\") "
Oct 30 06:21:49.411721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411631    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-socket" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:49.411855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.411837    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-data" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:49.412427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.412398    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.412606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.412582    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.412730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.412608    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.412730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.412624    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.412947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.412926    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e377ba94-aefe-4012-b1eb-363b2ba740d8-istiod-ca-cert" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:21:49.414011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.413976    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-kube-api-access-zqn4x" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "kube-api-access-zqn4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:49.414450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.414429    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-token" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:49.414534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.414428    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-envoy" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:49.415338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.415320    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-podinfo" pod "e377ba94-aefe-4012-b1eb-363b2ba740d8" (UID: "e377ba94-aefe-4012-b1eb-363b2ba740d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:21:49.444610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.444578    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"]
Oct 30 06:21:49.513951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.513920    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.513951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.513949    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.514134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.513958    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377ba94-aefe-4012-b1eb-363b2ba740d8-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.514134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.513967    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377ba94-aefe-4012-b1eb-363b2ba740d8-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.514134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.513975    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqn4x\" (UniqueName: \"kubernetes.io/projected/e377ba94-aefe-4012-b1eb-363b2ba740d8-kube-api-access-zqn4x\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:49.570456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.570360    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:49.796740 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.796710    5250 generic.go:356] "Generic (PLEG): container finished" podID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerID="f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e" exitCode=0
Oct 30 06:21:49.797237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.796764    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"
Oct 30 06:21:49.797237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.796806    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" event={"ID":"e377ba94-aefe-4012-b1eb-363b2ba740d8","Type":"ContainerDied","Data":"f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e"}
Oct 30 06:21:49.797237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.796863    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z" event={"ID":"e377ba94-aefe-4012-b1eb-363b2ba740d8","Type":"ContainerDied","Data":"553de14e3ba7f6cc76208dc07ac75ba4f2e59941391d519a948054d0a1cecea6"}
Oct 30 06:21:49.797237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.796885    5250 scope.go:122] "RemoveContainer" containerID="f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e"
Oct 30 06:21:49.809110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.809092    5250 scope.go:122] "RemoveContainer" containerID="7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75"
Oct 30 06:21:49.820540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.820488    5250 scope.go:122] "RemoveContainer" containerID="f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e"
Oct 30 06:21:49.820857 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:49.820827    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e\": container with ID starting with f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e not found: ID does not exist" containerID="f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e"
Oct 30 06:21:49.820956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.820872    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e"} err="failed to get container status \"f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e\": rpc error: code = NotFound desc = could not find container \"f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e\": container with ID starting with f8ff2b384573e861d57c5e53c560a3677bd6349206dcf8559bf0fd566d3a782e not found: ID does not exist"
Oct 30 06:21:49.820956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.820897    5250 scope.go:122] "RemoveContainer" containerID="7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75"
Oct 30 06:21:49.821251 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:49.821157    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75\": container with ID starting with 7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75 not found: ID does not exist" containerID="7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75"
Oct 30 06:21:49.821304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.821240    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75"} err="failed to get container status \"7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75\": rpc error: code = NotFound desc = could not find container \"7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75\": container with ID starting with 7e6e8b492a0a9149a01c78971770095eead57327c1bb9527670c580fde7bbd75 not found: ID does not exist"
Oct 30 06:21:49.844868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.844837    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"]
Oct 30 06:21:49.848316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.848288    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-xfg9z"]
Oct 30 06:21:49.943046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:49.943021    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:50.020238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.020202    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29e4618-4c76-4223-ad25-0a24881d1eb4-kube-api-access-v4zvz\" (UniqueName: \"kubernetes.io/projected/b29e4618-4c76-4223-ad25-0a24881d1eb4-kube-api-access-v4zvz\") pod \"b29e4618-4c76-4223-ad25-0a24881d1eb4\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") "
Oct 30 06:21:50.020462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.020442    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-xks-ca\") pod \"b29e4618-4c76-4223-ad25-0a24881d1eb4\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") "
Oct 30 06:21:50.020563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.020545    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-openshift-ca\") pod \"b29e4618-4c76-4223-ad25-0a24881d1eb4\" (UID: \"b29e4618-4c76-4223-ad25-0a24881d1eb4\") "
Oct 30 06:21:50.020860 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.020839    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-openshift-ca" pod "b29e4618-4c76-4223-ad25-0a24881d1eb4" (UID: "b29e4618-4c76-4223-ad25-0a24881d1eb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:21:50.021087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.021071    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:50.022738 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.022716    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-xks-ca" pod "b29e4618-4c76-4223-ad25-0a24881d1eb4" (UID: "b29e4618-4c76-4223-ad25-0a24881d1eb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:21:50.023063 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.023049    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29e4618-4c76-4223-ad25-0a24881d1eb4-kube-api-access-v4zvz" pod "b29e4618-4c76-4223-ad25-0a24881d1eb4" (UID: "b29e4618-4c76-4223-ad25-0a24881d1eb4"). InnerVolumeSpecName "kube-api-access-v4zvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:50.122603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.122516    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29e4618-4c76-4223-ad25-0a24881d1eb4-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:50.122603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.122545    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zvz\" (UniqueName: \"kubernetes.io/projected/b29e4618-4c76-4223-ad25-0a24881d1eb4-kube-api-access-v4zvz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:50.215809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.215776    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:50.805336 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.805306    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"
Oct 30 06:21:50.805780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.805339    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r" event={"ID":"b29e4618-4c76-4223-ad25-0a24881d1eb4","Type":"ContainerDied","Data":"5c9f6e4bf76f5c0b2a40d19213739908ceced97016eda8e71b17062b0e2fb513"}
Oct 30 06:21:50.805780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:50.805400    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c9f6e4bf76f5c0b2a40d19213739908ceced97016eda8e71b17062b0e2fb513"
Oct 30 06:21:51.282828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:51.282794    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" path="/var/lib/kubelet/pods/e377ba94-aefe-4012-b1eb-363b2ba740d8/volumes"
Oct 30 06:21:51.349778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:51.349749    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:51.494446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:51.494411    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:51.687954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:51.687916    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:52.088535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.088455    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"]
Oct 30 06:21:52.089087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.089071    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy"
Oct 30 06:21:52.089155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.089090    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy"
Oct 30 06:21:52.089155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.089112    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b29e4618-4c76-4223-ad25-0a24881d1eb4" containerName="revoke-keys"
Oct 30 06:21:52.089155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.089117    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29e4618-4c76-4223-ad25-0a24881d1eb4" containerName="revoke-keys"
Oct 30 06:21:52.089155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.089127    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy"
Oct 30 06:21:52.089155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.089132    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy"
Oct 30 06:21:52.089398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.089229    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy"
Oct 30 06:21:52.089398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.089240    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29e4618-4c76-4223-ad25-0a24881d1eb4" containerName="revoke-keys"
Oct 30 06:21:52.093891 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.093873    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.110062 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.110035    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"]
Oct 30 06:21:52.115622 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.115599    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:21:52.115725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.115606    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:21:52.256766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.256728    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.256766 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.256766    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjp2\" (UniqueName: \"kubernetes.io/projected/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-kube-api-access-6kjp2\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.256988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.256792    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.357970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.357867    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.357970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.357909    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjp2\" (UniqueName: \"kubernetes.io/projected/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-kube-api-access-6kjp2\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.357970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.357953    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.358703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.358679    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.360732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.360712    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.379272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.379246    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjp2\" (UniqueName: \"kubernetes.io/projected/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-kube-api-access-6kjp2\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.408154 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.408128    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:52.665104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.665065    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:52.714079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.714007    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"]
Oct 30 06:21:52.717186 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:21:52.717148    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4899d4a4_27cc_4efa_a9ea_0017342bb0fc.slice/crio-d50734472cda83d00c81eaebaed044e8b5f5b24ebaf893889f37905dfcf00319 WatchSource:0}: Error finding container d50734472cda83d00c81eaebaed044e8b5f5b24ebaf893889f37905dfcf00319: Status 404 returned error can't find the container with id d50734472cda83d00c81eaebaed044e8b5f5b24ebaf893889f37905dfcf00319
Oct 30 06:21:52.832948 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:52.832914    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d" event={"ID":"4899d4a4-27cc-4efa-a9ea-0017342bb0fc","Type":"ContainerStarted","Data":"d50734472cda83d00c81eaebaed044e8b5f5b24ebaf893889f37905dfcf00319"}
Oct 30 06:21:53.841570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:53.841530    5250 generic.go:356] "Generic (PLEG): container finished" podID="4899d4a4-27cc-4efa-a9ea-0017342bb0fc" containerID="cfe2452154149ecdfa473f18b3d11ee69031cd4e1987bc66e1404ea5e3729a2c" exitCode=0
Oct 30 06:21:53.842004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:53.841614    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d" event={"ID":"4899d4a4-27cc-4efa-a9ea-0017342bb0fc","Type":"ContainerDied","Data":"cfe2452154149ecdfa473f18b3d11ee69031cd4e1987bc66e1404ea5e3729a2c"}
Oct 30 06:21:54.277025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.276964    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:21:54.384185 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384155    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-envoy\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384257    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-certs\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384316    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-token\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-token\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384463 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384383    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-podinfo\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384466    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-credential-socket\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384540    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723e1a96-a612-4c85-9dbe-1d71d08938ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723e1a96-a612-4c85-9dbe-1d71d08938ba-istiod-ca-cert\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384544    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-certs" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:54.384672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384575    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-data\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384611    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-socket\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384653    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-kube-api-access-2lq5x\") pod \"723e1a96-a612-4c85-9dbe-1d71d08938ba\" (UID: \"723e1a96-a612-4c85-9dbe-1d71d08938ba\") "
Oct 30 06:21:54.384986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384940    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-socket" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:54.384986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384958    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-credential-socket" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:54.385148 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.385014    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723e1a96-a612-4c85-9dbe-1d71d08938ba-istiod-ca-cert" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:21:54.385148 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.384896    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-data" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:54.385617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.385525    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.385617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.385551    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723e1a96-a612-4c85-9dbe-1d71d08938ba-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.385617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.385565    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.385617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.385577    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.385617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.385591    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.386765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.386740    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-envoy" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:21:54.386908 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.386889    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-podinfo" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:21:54.387339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.387320    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-token" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:54.387495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.387475    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-kube-api-access-2lq5x" pod "723e1a96-a612-4c85-9dbe-1d71d08938ba" (UID: "723e1a96-a612-4c85-9dbe-1d71d08938ba"). InnerVolumeSpecName "kube-api-access-2lq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:54.486833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.486803    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.486833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.486829    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.487002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.486841    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723e1a96-a612-4c85-9dbe-1d71d08938ba-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.487002 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.486855    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/723e1a96-a612-4c85-9dbe-1d71d08938ba-kube-api-access-2lq5x\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:54.666300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.666267    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:54.851758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.851724    5250 generic.go:356] "Generic (PLEG): container finished" podID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerID="5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7" exitCode=0
Oct 30 06:21:54.852235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.851775    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"
Oct 30 06:21:54.852235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.851808    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" event={"ID":"723e1a96-a612-4c85-9dbe-1d71d08938ba","Type":"ContainerDied","Data":"5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7"}
Oct 30 06:21:54.852235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.851863    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4" event={"ID":"723e1a96-a612-4c85-9dbe-1d71d08938ba","Type":"ContainerDied","Data":"10d766e3fb3696a065eea6ecda9198916c702025007e5d296460f495d12c9db2"}
Oct 30 06:21:54.852235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.851889    5250 scope.go:122] "RemoveContainer" containerID="5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7"
Oct 30 06:21:54.865632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.865607    5250 scope.go:122] "RemoveContainer" containerID="8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b"
Oct 30 06:21:54.878241 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.878218    5250 scope.go:122] "RemoveContainer" containerID="5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7"
Oct 30 06:21:54.878542 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:54.878520    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7\": container with ID starting with 5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7 not found: ID does not exist" containerID="5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7"
Oct 30 06:21:54.878632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.878552    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7"} err="failed to get container status \"5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7\": rpc error: code = NotFound desc = could not find container \"5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7\": container with ID starting with 5aa697e31ea555d1aca6b0d299f9af4de220d5927f5b5aae6f32098b314f14b7 not found: ID does not exist"
Oct 30 06:21:54.878632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.878569    5250 scope.go:122] "RemoveContainer" containerID="8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b"
Oct 30 06:21:54.878863 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:21:54.878843    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b\": container with ID starting with 8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b not found: ID does not exist" containerID="8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b"
Oct 30 06:21:54.878913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.878868    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b"} err="failed to get container status \"8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b\": rpc error: code = NotFound desc = could not find container \"8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b\": container with ID starting with 8e55b1d979dd48af9e49fc48eeeafe07ca2ff7edd69b48197f31f3d93dcdfd8b not found: ID does not exist"
Oct 30 06:21:54.917107 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.917037    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"]
Oct 30 06:21:54.929220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:54.928475    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a591d-openshift-default-7c5664f79-prnf4"]
Oct 30 06:21:55.012133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.012101    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"]
Oct 30 06:21:55.019043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.019019    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:55.174495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.174400    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:55.196015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.195986    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-xks-ca\") pod \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") "
Oct 30 06:21:55.196142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.196096    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-openshift-ca\") pod \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") "
Oct 30 06:21:55.196142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.196120    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-kube-api-access-6kjp2\" (UniqueName: \"kubernetes.io/projected/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-kube-api-access-6kjp2\") pod \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\" (UID: \"4899d4a4-27cc-4efa-a9ea-0017342bb0fc\") "
Oct 30 06:21:55.196555 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.196530    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-openshift-ca" pod "4899d4a4-27cc-4efa-a9ea-0017342bb0fc" (UID: "4899d4a4-27cc-4efa-a9ea-0017342bb0fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:21:55.198244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.198222    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-xks-ca" pod "4899d4a4-27cc-4efa-a9ea-0017342bb0fc" (UID: "4899d4a4-27cc-4efa-a9ea-0017342bb0fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:21:55.198482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.198459    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-kube-api-access-6kjp2" pod "4899d4a4-27cc-4efa-a9ea-0017342bb0fc" (UID: "4899d4a4-27cc-4efa-a9ea-0017342bb0fc"). InnerVolumeSpecName "kube-api-access-6kjp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:21:55.282951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.282909    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" path="/var/lib/kubelet/pods/723e1a96-a612-4c85-9dbe-1d71d08938ba/volumes"
Oct 30 06:21:55.297436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.297402    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:55.297436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.297433    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:55.297616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.297450    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjp2\" (UniqueName: \"kubernetes.io/projected/4899d4a4-27cc-4efa-a9ea-0017342bb0fc-kube-api-access-6kjp2\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:21:55.322215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.322160    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:55.860109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.860081    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"
Oct 30 06:21:55.860109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.860095    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d" event={"ID":"4899d4a4-27cc-4efa-a9ea-0017342bb0fc","Type":"ContainerDied","Data":"d50734472cda83d00c81eaebaed044e8b5f5b24ebaf893889f37905dfcf00319"}
Oct 30 06:21:55.860648 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:55.860166    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d50734472cda83d00c81eaebaed044e8b5f5b24ebaf893889f37905dfcf00319"
Oct 30 06:21:56.280052 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:56.280001    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:57.803642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:57.803596    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:59.582035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:59.582000    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:21:59.897806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:21:59.897771    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"]
Oct 30 06:22:00.012820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:00.012777    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w" podUID="7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" containerName="payload-processing" containerID="cri-o://75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d" gracePeriod=25
Oct 30 06:22:00.022801 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:00.022771    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:00.166146 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:00.166058    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:01.248950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:01.248915    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:02.374571 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:22:02.374533    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:22:03.506281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:03.506246    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:04.467757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.467713    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main" containerID="cri-o://46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1" gracePeriod=45
Oct 30 06:22:04.635380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.635349    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:04.789460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.780023    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:04.801070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.801046    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 06:22:04.830359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830321    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tls-certs\") pod \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") "
Oct 30 06:22:04.830513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830388    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-kube-api-access-kfsl8\" (UniqueName: \"kubernetes.io/projected/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-kube-api-access-kfsl8\") pod \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") "
Oct 30 06:22:04.830513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830434    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-home\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-home\") pod \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") "
Oct 30 06:22:04.830642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830579    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-dshm\") pod \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") "
Oct 30 06:22:04.830713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830697    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-model-cache\") pod \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") "
Oct 30 06:22:04.830775 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830728    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tmp-dir\") pod \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\" (UID: \"9c0f2cb3-53e8-4162-aefe-3b99db4ba678\") "
Oct 30 06:22:04.830854 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830726    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-home" pod "9c0f2cb3-53e8-4162-aefe-3b99db4ba678" (UID: "9c0f2cb3-53e8-4162-aefe-3b99db4ba678"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:04.830920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830901    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tmp-dir" pod "9c0f2cb3-53e8-4162-aefe-3b99db4ba678" (UID: "9c0f2cb3-53e8-4162-aefe-3b99db4ba678"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:04.830976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.830934    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-model-cache" pod "9c0f2cb3-53e8-4162-aefe-3b99db4ba678" (UID: "9c0f2cb3-53e8-4162-aefe-3b99db4ba678"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:04.831329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.831311    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:04.831422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.831331    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:04.831422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.831345    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:04.832945 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.832925    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-dshm" pod "9c0f2cb3-53e8-4162-aefe-3b99db4ba678" (UID: "9c0f2cb3-53e8-4162-aefe-3b99db4ba678"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:04.833016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.832962    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-kube-api-access-kfsl8" pod "9c0f2cb3-53e8-4162-aefe-3b99db4ba678" (UID: "9c0f2cb3-53e8-4162-aefe-3b99db4ba678"). InnerVolumeSpecName "kube-api-access-kfsl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:04.833016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.832961    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tls-certs" pod "9c0f2cb3-53e8-4162-aefe-3b99db4ba678" (UID: "9c0f2cb3-53e8-4162-aefe-3b99db4ba678"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:22:04.898162 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.898113    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j" podUID="41fbcd02-5042-4a31-9903-ac370777dec3" containerName="payload-pre-processing" containerID="cri-o://fedd44354a9aed0e1780d915b8aef565efaa184a0e2382d21a28a9b1524d55ce" gracePeriod=25
Oct 30 06:22:04.930297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.930264    5250 generic.go:356] "Generic (PLEG): container finished" podID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerID="46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1" exitCode=0
Oct 30 06:22:04.930488 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.930318    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"
Oct 30 06:22:04.930488 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.930448    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" event={"ID":"9c0f2cb3-53e8-4162-aefe-3b99db4ba678","Type":"ContainerDied","Data":"46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1"}
Oct 30 06:22:04.930488 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.930486    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh" event={"ID":"9c0f2cb3-53e8-4162-aefe-3b99db4ba678","Type":"ContainerDied","Data":"1d7fea01b5f4505d0086a805235366054b9fb48ecf5cd9fd57b7d28dc29db4c7"}
Oct 30 06:22:04.930674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.930507    5250 scope.go:122] "RemoveContainer" containerID="46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1"
Oct 30 06:22:04.940101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.940070    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:04.940275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.940104    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfsl8\" (UniqueName: \"kubernetes.io/projected/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-kube-api-access-kfsl8\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:04.940275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.940119    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0f2cb3-53e8-4162-aefe-3b99db4ba678-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:04.966983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.966948    5250 scope.go:122] "RemoveContainer" containerID="46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1"
Oct 30 06:22:04.967535 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:22:04.967502    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1\": container with ID starting with 46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1 not found: ID does not exist" containerID="46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1"
Oct 30 06:22:04.967657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.967546    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1"} err="failed to get container status \"46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1\": rpc error: code = NotFound desc = could not find container \"46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1\": container with ID starting with 46457a6564ec8c7f52cae701478600710c578a46120d449bad362ad188f4ceb1 not found: ID does not exist"
Oct 30 06:22:04.987353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.987322    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"]
Oct 30 06:22:04.995843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:04.995817    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/premium-w0-a14091-kserve-78b6ffb4f4-mfjsh"]
Oct 30 06:22:05.281503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:05.281474    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" path="/var/lib/kubelet/pods/9c0f2cb3-53e8-4162-aefe-3b99db4ba678/volumes"
Oct 30 06:22:05.709079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:05.709039    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:07.203410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:07.203380    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:07.364760 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:07.364706    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:07.511309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:07.511223    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:07.677477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:07.677441    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w" podUID="7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:22:09.577905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:09.577870    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:11.161310 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:11.161278    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:12.709443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:12.709408    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:14.000109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:14.000072    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j" podUID="41fbcd02-5042-4a31-9903-ac370777dec3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:22:14.373067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:14.372985    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:15.894083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:15.894047    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:17.464738 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:17.464152    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:17.677162 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:17.677118    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w" podUID="7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:22:19.407667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.407635    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.767331    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"]
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.767917    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.767928    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.767956    5250 cpu_manager.go:414] "containerMap: removing container" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.767961    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.767984    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4899d4a4-27cc-4efa-a9ea-0017342bb0fc" containerName="revoke-keys"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.767990    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4899d4a4-27cc-4efa-a9ea-0017342bb0fc" containerName="revoke-keys"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.768075    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.768086    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4899d4a4-27cc-4efa-a9ea-0017342bb0fc" containerName="revoke-keys"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.768094    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e377ba94-aefe-4012-b1eb-363b2ba740d8" containerName="istio-proxy"
Oct 30 06:22:19.768256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.768103    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0f2cb3-53e8-4162-aefe-3b99db4ba678" containerName="main"
Oct 30 06:22:19.776279 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.773225    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.814500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.814416    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:22:19.815501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.815469    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:22:19.829100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.818531    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"]
Oct 30 06:22:19.830446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.830408    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"]
Oct 30 06:22:19.831653 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.831627    5250 cpu_manager.go:414] "containerMap: removing container" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy"
Oct 30 06:22:19.831799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.831784    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy"
Oct 30 06:22:19.832120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.832101    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="723e1a96-a612-4c85-9dbe-1d71d08938ba" containerName="istio-proxy"
Oct 30 06:22:19.836427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.836407    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"]
Oct 30 06:22:19.837307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.836617    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.866694 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.866662    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.866920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.866834    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r68\" (UniqueName: \"kubernetes.io/projected/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-kube-api-access-67r68\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.867035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.867011    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.867103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.867091    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.867164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.867151    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.867248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.867199    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4w4x\" (UniqueName: \"kubernetes.io/projected/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-kube-api-access-c4w4x\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.968068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.968035    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67r68\" (UniqueName: \"kubernetes.io/projected/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-kube-api-access-67r68\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.968291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.968131    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.968291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.968166    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.968291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.968232    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.968291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.968252    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4w4x\" (UniqueName: \"kubernetes.io/projected/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-kube-api-access-c4w4x\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.968291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.968277    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.968943 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.968920    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.969037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.968924    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.970830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.970803    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:19.970934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.970844    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.989041 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.989012    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4w4x\" (UniqueName: \"kubernetes.io/projected/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-kube-api-access-c4w4x\") pod \"maas-api-revoke-sub-ac7da157-fn59d\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:19.989253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:19.989232    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r68\" (UniqueName: \"kubernetes.io/projected/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-kube-api-access-67r68\") pod \"maas-api-revoke-sub-9f664c86-l2xxd\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:20.006898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.006873    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:20.125331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.125303    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:20.157528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.157501    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:20.241138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.240585    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:20.336111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.336078    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 06:22:20.376010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.375973    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-kube-api-access-qcdx6\" (UniqueName: \"kubernetes.io/projected/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-kube-api-access-qcdx6\") pod \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\" (UID: \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\") "
Oct 30 06:22:20.376235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.376142    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-plugins-config-volume\") pod \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\" (UID: \"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02\") "
Oct 30 06:22:20.376561 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.376533    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-plugins-config-volume" pod "7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" (UID: "7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:20.376807 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.376788    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:20.378696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.378674    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-kube-api-access-qcdx6" pod "7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" (UID: "7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02"). InnerVolumeSpecName "kube-api-access-qcdx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:20.425759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.425730    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:20.445319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.444405    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"]
Oct 30 06:22:20.448012 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:22:20.447972    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1ed1aa_5d0d_44ea_a879_4451d82665dd.slice/crio-1af5bc2ce57ed366dfd21a958cd1e231bd18d27b91ecddf0519d91a610e51a95 WatchSource:0}: Error finding container 1af5bc2ce57ed366dfd21a958cd1e231bd18d27b91ecddf0519d91a610e51a95: Status 404 returned error can't find the container with id 1af5bc2ce57ed366dfd21a958cd1e231bd18d27b91ecddf0519d91a610e51a95
Oct 30 06:22:20.478099 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.478068    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcdx6\" (UniqueName: \"kubernetes.io/projected/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02-kube-api-access-qcdx6\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:20.485314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.485285    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"]
Oct 30 06:22:20.488648 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:22:20.488624    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbcf479_cb5e_4a22_a613_9cc66b6803e6.slice/crio-845cc15562b4ebd617a1ca58cfd8fd85facb05c84a21d5628b2a51e7598f3d68 WatchSource:0}: Error finding container 845cc15562b4ebd617a1ca58cfd8fd85facb05c84a21d5628b2a51e7598f3d68: Status 404 returned error can't find the container with id 845cc15562b4ebd617a1ca58cfd8fd85facb05c84a21d5628b2a51e7598f3d68
Oct 30 06:22:20.810378 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:20.810337    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:21.051844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.051806    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d" event={"ID":"9f1ed1aa-5d0d-44ea-a879-4451d82665dd","Type":"ContainerStarted","Data":"4bcf721e0a579eacdba07d95fdcfee1c33dc3453c8e66e7d1bc01b1d0b195f5c"}
Oct 30 06:22:21.052016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.051865    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d" event={"ID":"9f1ed1aa-5d0d-44ea-a879-4451d82665dd","Type":"ContainerStarted","Data":"1af5bc2ce57ed366dfd21a958cd1e231bd18d27b91ecddf0519d91a610e51a95"}
Oct 30 06:22:21.053234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.053209    5250 generic.go:356] "Generic (PLEG): container finished" podID="7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" containerID="75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d" exitCode=0
Oct 30 06:22:21.053372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.053245    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"
Oct 30 06:22:21.053372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.053283    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w" event={"ID":"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02","Type":"ContainerDied","Data":"75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d"}
Oct 30 06:22:21.053372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.053331    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w" event={"ID":"7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02","Type":"ContainerDied","Data":"3c2a4286912e6c8c4df72f040940f24eff885855fa0a7eb126fc6cc0b759dc70"}
Oct 30 06:22:21.053372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.053356    5250 scope.go:122] "RemoveContainer" containerID="75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d"
Oct 30 06:22:21.055358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.055329    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd" event={"ID":"1dbcf479-cb5e-4a22-a613-9cc66b6803e6","Type":"ContainerStarted","Data":"4eeabc51b4320229ea8a2a2f818a9dd0390bac16a298815f01c21be7a469ab68"}
Oct 30 06:22:21.055475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.055385    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd" event={"ID":"1dbcf479-cb5e-4a22-a613-9cc66b6803e6","Type":"ContainerStarted","Data":"845cc15562b4ebd617a1ca58cfd8fd85facb05c84a21d5628b2a51e7598f3d68"}
Oct 30 06:22:21.066057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.066039    5250 scope.go:122] "RemoveContainer" containerID="75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d"
Oct 30 06:22:21.066360 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:22:21.066332    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d\": container with ID starting with 75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d not found: ID does not exist" containerID="75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d"
Oct 30 06:22:21.066462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.066365    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d"} err="failed to get container status \"75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d\": rpc error: code = NotFound desc = could not find container \"75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d\": container with ID starting with 75da60c9ec116cb3e258583c31b4232888869e1f443fe3586a42d50314ae5b2d not found: ID does not exist"
Oct 30 06:22:21.091904 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.091857    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d" podStartSLOduration=2.091844044 podStartE2EDuration="2.091844044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:21.088399015 +0000 UTC m=+3055.919436233" watchObservedRunningTime="2026-10-30 06:22:21.091844044 +0000 UTC m=+3055.922881318"
Oct 30 06:22:21.121700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.121666    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"]
Oct 30 06:22:21.126368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.126329    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a14091-bd458df46-zp84w"]
Oct 30 06:22:21.157675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.157616    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd" podStartSLOduration=2.157597227 podStartE2EDuration="2.157597227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:21.156857617 +0000 UTC m=+3055.987894834" watchObservedRunningTime="2026-10-30 06:22:21.157597227 +0000 UTC m=+3055.988634445"
Oct 30 06:22:21.282629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:21.282598    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" path="/var/lib/kubelet/pods/7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02/volumes"
Oct 30 06:22:22.031638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.031598    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:22.064904 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.064868    5250 generic.go:356] "Generic (PLEG): container finished" podID="1dbcf479-cb5e-4a22-a613-9cc66b6803e6" containerID="4eeabc51b4320229ea8a2a2f818a9dd0390bac16a298815f01c21be7a469ab68" exitCode=0
Oct 30 06:22:22.065101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.064947    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd" event={"ID":"1dbcf479-cb5e-4a22-a613-9cc66b6803e6","Type":"ContainerDied","Data":"4eeabc51b4320229ea8a2a2f818a9dd0390bac16a298815f01c21be7a469ab68"}
Oct 30 06:22:22.066894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.066872    5250 generic.go:356] "Generic (PLEG): container finished" podID="9f1ed1aa-5d0d-44ea-a879-4451d82665dd" containerID="4bcf721e0a579eacdba07d95fdcfee1c33dc3453c8e66e7d1bc01b1d0b195f5c" exitCode=0
Oct 30 06:22:22.067050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.066959    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d" event={"ID":"9f1ed1aa-5d0d-44ea-a879-4451d82665dd","Type":"ContainerDied","Data":"4bcf721e0a579eacdba07d95fdcfee1c33dc3453c8e66e7d1bc01b1d0b195f5c"}
Oct 30 06:22:22.219445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.219410    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:22.360614 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.360497    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:22.581951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.581900    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:22.717410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.717369    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"]
Oct 30 06:22:22.720809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.720784    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07b278-hvq2r"]
Oct 30 06:22:22.728860 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.728834    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:22.874247 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:22.874110    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:23.233562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.233533    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:23.250546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.250510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:23.282498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.282474    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:23.283872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.283842    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29e4618-4c76-4223-ad25-0a24881d1eb4" path="/var/lib/kubelet/pods/b29e4618-4c76-4223-ad25-0a24881d1eb4/volumes"
Oct 30 06:22:23.322616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.322583    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-xks-ca\") pod \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") "
Oct 30 06:22:23.322852 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.322675    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-kube-api-access-67r68\" (UniqueName: \"kubernetes.io/projected/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-kube-api-access-67r68\") pod \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") "
Oct 30 06:22:23.322852 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.322774    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-openshift-ca\") pod \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") "
Oct 30 06:22:23.323019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.322912    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-xks-ca\") pod \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") "
Oct 30 06:22:23.323019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.322949    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-kube-api-access-c4w4x\" (UniqueName: \"kubernetes.io/projected/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-kube-api-access-c4w4x\") pod \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\" (UID: \"9f1ed1aa-5d0d-44ea-a879-4451d82665dd\") "
Oct 30 06:22:23.323149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.323057    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-openshift-ca\") pod \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\" (UID: \"1dbcf479-cb5e-4a22-a613-9cc66b6803e6\") "
Oct 30 06:22:23.323416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.323390    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-openshift-ca" pod "9f1ed1aa-5d0d-44ea-a879-4451d82665dd" (UID: "9f1ed1aa-5d0d-44ea-a879-4451d82665dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:23.323737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.323710    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-openshift-ca" pod "1dbcf479-cb5e-4a22-a613-9cc66b6803e6" (UID: "1dbcf479-cb5e-4a22-a613-9cc66b6803e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:23.323963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.323944    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:23.324051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.324018    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:23.326209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.325727    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-xks-ca" pod "1dbcf479-cb5e-4a22-a613-9cc66b6803e6" (UID: "1dbcf479-cb5e-4a22-a613-9cc66b6803e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:22:23.326570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.326544    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-kube-api-access-c4w4x" pod "9f1ed1aa-5d0d-44ea-a879-4451d82665dd" (UID: "9f1ed1aa-5d0d-44ea-a879-4451d82665dd"). InnerVolumeSpecName "kube-api-access-c4w4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:23.326812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.326795    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-kube-api-access-67r68" pod "1dbcf479-cb5e-4a22-a613-9cc66b6803e6" (UID: "1dbcf479-cb5e-4a22-a613-9cc66b6803e6"). InnerVolumeSpecName "kube-api-access-67r68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:23.326907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.326869    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-xks-ca" pod "9f1ed1aa-5d0d-44ea-a879-4451d82665dd" (UID: "9f1ed1aa-5d0d-44ea-a879-4451d82665dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:22:23.425060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.425022    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:23.425060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.425059    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67r68\" (UniqueName: \"kubernetes.io/projected/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-kube-api-access-67r68\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:23.425060 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.425070    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf479-cb5e-4a22-a613-9cc66b6803e6-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:23.425314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.425078    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4w4x\" (UniqueName: \"kubernetes.io/projected/9f1ed1aa-5d0d-44ea-a879-4451d82665dd-kube-api-access-c4w4x\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:23.999945 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:23.999909    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j" podUID="41fbcd02-5042-4a31-9903-ac370777dec3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:22:24.083326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:24.083296    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"
Oct 30 06:22:24.083513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:24.083338    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd" event={"ID":"1dbcf479-cb5e-4a22-a613-9cc66b6803e6","Type":"ContainerDied","Data":"845cc15562b4ebd617a1ca58cfd8fd85facb05c84a21d5628b2a51e7598f3d68"}
Oct 30 06:22:24.083513 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:24.083409    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="845cc15562b4ebd617a1ca58cfd8fd85facb05c84a21d5628b2a51e7598f3d68"
Oct 30 06:22:24.084894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:24.084874    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"
Oct 30 06:22:24.085019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:24.084917    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d" event={"ID":"9f1ed1aa-5d0d-44ea-a879-4451d82665dd","Type":"ContainerDied","Data":"1af5bc2ce57ed366dfd21a958cd1e231bd18d27b91ecddf0519d91a610e51a95"}
Oct 30 06:22:24.085019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:24.084957    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1af5bc2ce57ed366dfd21a958cd1e231bd18d27b91ecddf0519d91a610e51a95"
Oct 30 06:22:24.713277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:24.713233    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:25.099448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.099416    5250 generic.go:356] "Generic (PLEG): container finished" podID="41fbcd02-5042-4a31-9903-ac370777dec3" containerID="fedd44354a9aed0e1780d915b8aef565efaa184a0e2382d21a28a9b1524d55ce" exitCode=0
Oct 30 06:22:25.099651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.099549    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j" event={"ID":"41fbcd02-5042-4a31-9903-ac370777dec3","Type":"ContainerDied","Data":"fedd44354a9aed0e1780d915b8aef565efaa184a0e2382d21a28a9b1524d55ce"}
Oct 30 06:22:25.175375 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.175345    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:22:25.255399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.255362    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fbcd02-5042-4a31-9903-ac370777dec3-kube-api-access-mhjpf\" (UniqueName: \"kubernetes.io/projected/41fbcd02-5042-4a31-9903-ac370777dec3-kube-api-access-mhjpf\") pod \"41fbcd02-5042-4a31-9903-ac370777dec3\" (UID: \"41fbcd02-5042-4a31-9903-ac370777dec3\") "
Oct 30 06:22:25.255603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.255453    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41fbcd02-5042-4a31-9903-ac370777dec3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fbcd02-5042-4a31-9903-ac370777dec3-plugins-config-volume\") pod \"41fbcd02-5042-4a31-9903-ac370777dec3\" (UID: \"41fbcd02-5042-4a31-9903-ac370777dec3\") "
Oct 30 06:22:25.255873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.255842    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fbcd02-5042-4a31-9903-ac370777dec3-plugins-config-volume" pod "41fbcd02-5042-4a31-9903-ac370777dec3" (UID: "41fbcd02-5042-4a31-9903-ac370777dec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:25.257696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.257675    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fbcd02-5042-4a31-9903-ac370777dec3-kube-api-access-mhjpf" pod "41fbcd02-5042-4a31-9903-ac370777dec3" (UID: "41fbcd02-5042-4a31-9903-ac370777dec3"). InnerVolumeSpecName "kube-api-access-mhjpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:25.356544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.356511    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhjpf\" (UniqueName: \"kubernetes.io/projected/41fbcd02-5042-4a31-9903-ac370777dec3-kube-api-access-mhjpf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:25.356544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:25.356537    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fbcd02-5042-4a31-9903-ac370777dec3-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:26.109230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:26.109201    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"
Oct 30 06:22:26.109230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:26.109218    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j" event={"ID":"41fbcd02-5042-4a31-9903-ac370777dec3","Type":"ContainerDied","Data":"dba5ccf62f505daf56df29c6b938cbe78527e4d9ac3efc62a7af1f16a9d69f54"}
Oct 30 06:22:26.109753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:26.109287    5250 scope.go:122] "RemoveContainer" containerID="fedd44354a9aed0e1780d915b8aef565efaa184a0e2382d21a28a9b1524d55ce"
Oct 30 06:22:26.159370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:26.159345    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"]
Oct 30 06:22:26.169280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:26.169256    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-080391-5df5774c-xld9j"]
Oct 30 06:22:26.419558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:26.419510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:26.564401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:26.564332    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:26.713274 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:26.706966    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:27.281399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:27.281370    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41fbcd02-5042-4a31-9903-ac370777dec3" path="/var/lib/kubelet/pods/41fbcd02-5042-4a31-9903-ac370777dec3/volumes"
Oct 30 06:22:28.054869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.051532    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:28.263481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.263451    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"]
Oct 30 06:22:28.264398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.264363    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy" containerID="cri-o://d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da" gracePeriod=30
Oct 30 06:22:28.287736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.287705    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:28.435456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.435423    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:28.825454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.825368    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"]
Oct 30 06:22:28.826157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826134    5250 cpu_manager.go:414] "containerMap: removing container" podUID="7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" containerName="payload-processing"
Oct 30 06:22:28.826276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826160    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" containerName="payload-processing"
Oct 30 06:22:28.826276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826193    5250 cpu_manager.go:414] "containerMap: removing container" podUID="41fbcd02-5042-4a31-9903-ac370777dec3" containerName="payload-pre-processing"
Oct 30 06:22:28.826276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826202    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41fbcd02-5042-4a31-9903-ac370777dec3" containerName="payload-pre-processing"
Oct 30 06:22:28.826276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826227    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9f1ed1aa-5d0d-44ea-a879-4451d82665dd" containerName="revoke-keys"
Oct 30 06:22:28.826276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826234    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1ed1aa-5d0d-44ea-a879-4451d82665dd" containerName="revoke-keys"
Oct 30 06:22:28.826276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826268    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1dbcf479-cb5e-4a22-a613-9cc66b6803e6" containerName="revoke-keys"
Oct 30 06:22:28.826276 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826275    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbcf479-cb5e-4a22-a613-9cc66b6803e6" containerName="revoke-keys"
Oct 30 06:22:28.826517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826446    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbcf479-cb5e-4a22-a613-9cc66b6803e6" containerName="revoke-keys"
Oct 30 06:22:28.826517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826465    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2e1c1a-1396-4fe4-9a7e-f77f0b7fed02" containerName="payload-processing"
Oct 30 06:22:28.826517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826472    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="41fbcd02-5042-4a31-9903-ac370777dec3" containerName="payload-pre-processing"
Oct 30 06:22:28.826517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.826482    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1ed1aa-5d0d-44ea-a879-4451d82665dd" containerName="revoke-keys"
Oct 30 06:22:28.831332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.831312    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:28.847966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.847943    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"]
Oct 30 06:22:28.853957 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.853934    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:22:28.854113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.854062    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:22:28.934641 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.934605    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:28.934816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.934750    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbntp\" (UniqueName: \"kubernetes.io/projected/79337cdb-cbd2-441f-9011-e78a674f53a8-kube-api-access-tbntp\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:28.934873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:28.934854    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:29.036349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.036305    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:29.036514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.036414    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbntp\" (UniqueName: \"kubernetes.io/projected/79337cdb-cbd2-441f-9011-e78a674f53a8-kube-api-access-tbntp\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:29.036514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.036491    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:29.037051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.037028    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:29.039036 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.039008    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:29.057644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.057621    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbntp\" (UniqueName: \"kubernetes.io/projected/79337cdb-cbd2-441f-9011-e78a674f53a8-kube-api-access-tbntp\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:29.145286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.145245    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:29.307273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.307236    5250 scope.go:122] "RemoveContainer" containerID="afbf04757896fb87df7fc182e53d250e62dfbad9f92126351c491e186a56443a"
Oct 30 06:22:29.318101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.318078    5250 scope.go:122] "RemoveContainer" containerID="022e4e987551728719daa316fd372bdecce6adcf53bb88ec9b791802202bad50"
Oct 30 06:22:29.450449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:29.450428    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"]
Oct 30 06:22:29.453360 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:22:29.453328    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79337cdb_cbd2_441f_9011_e78a674f53a8.slice/crio-444bf52a2a4422eb7d34faad4e73e66eab1c4f8a3df5a2a72060a6db25187a22 WatchSource:0}: Error finding container 444bf52a2a4422eb7d34faad4e73e66eab1c4f8a3df5a2a72060a6db25187a22: Status 404 returned error can't find the container with id 444bf52a2a4422eb7d34faad4e73e66eab1c4f8a3df5a2a72060a6db25187a22
Oct 30 06:22:30.046971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:30.046862    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:30.145242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:30.145202    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5" event={"ID":"79337cdb-cbd2-441f-9011-e78a674f53a8","Type":"ContainerStarted","Data":"3a9b5349b1523e1ff6c6a573b5f03588e1596f83a74bc31b5aa462d101525ee0"}
Oct 30 06:22:30.145438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:30.145267    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5" event={"ID":"79337cdb-cbd2-441f-9011-e78a674f53a8","Type":"ContainerStarted","Data":"444bf52a2a4422eb7d34faad4e73e66eab1c4f8a3df5a2a72060a6db25187a22"}
Oct 30 06:22:30.183376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:30.183304    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5" podStartSLOduration=2.183287083 podStartE2EDuration="2.183287083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:30.182802978 +0000 UTC m=+3065.013840191" watchObservedRunningTime="2026-10-30 06:22:30.183287083 +0000 UTC m=+3065.014324301"
Oct 30 06:22:30.190708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:30.190684    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:30.345262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:30.345130    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:30.501821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:30.500843    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:31.154324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:31.154293    5250 generic.go:356] "Generic (PLEG): container finished" podID="79337cdb-cbd2-441f-9011-e78a674f53a8" containerID="3a9b5349b1523e1ff6c6a573b5f03588e1596f83a74bc31b5aa462d101525ee0" exitCode=0
Oct 30 06:22:31.154512 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:31.154383    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5" event={"ID":"79337cdb-cbd2-441f-9011-e78a674f53a8","Type":"ContainerDied","Data":"3a9b5349b1523e1ff6c6a573b5f03588e1596f83a74bc31b5aa462d101525ee0"}
Oct 30 06:22:31.854161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:31.854126    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:32.017356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.017316    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:32.159112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.159080    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:32.391210 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:22:32.391152    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:22:32.441536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.441506    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:32.492912 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.492877    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:32.610270 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.610236    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-openshift-ca\") pod \"79337cdb-cbd2-441f-9011-e78a674f53a8\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") "
Oct 30 06:22:32.610442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.610279    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79337cdb-cbd2-441f-9011-e78a674f53a8-kube-api-access-tbntp\" (UniqueName: \"kubernetes.io/projected/79337cdb-cbd2-441f-9011-e78a674f53a8-kube-api-access-tbntp\") pod \"79337cdb-cbd2-441f-9011-e78a674f53a8\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") "
Oct 30 06:22:32.610442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.610309    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-xks-ca\") pod \"79337cdb-cbd2-441f-9011-e78a674f53a8\" (UID: \"79337cdb-cbd2-441f-9011-e78a674f53a8\") "
Oct 30 06:22:32.610669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.610635    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-openshift-ca" pod "79337cdb-cbd2-441f-9011-e78a674f53a8" (UID: "79337cdb-cbd2-441f-9011-e78a674f53a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:32.610907 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.610886    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:32.612791 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.612762    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79337cdb-cbd2-441f-9011-e78a674f53a8-kube-api-access-tbntp" pod "79337cdb-cbd2-441f-9011-e78a674f53a8" (UID: "79337cdb-cbd2-441f-9011-e78a674f53a8"). InnerVolumeSpecName "kube-api-access-tbntp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:32.612979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.612814    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-xks-ca" pod "79337cdb-cbd2-441f-9011-e78a674f53a8" (UID: "79337cdb-cbd2-441f-9011-e78a674f53a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:22:32.615729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.615703    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"]
Oct 30 06:22:32.620566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.620543    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8b892-m755d"]
Oct 30 06:22:32.641767 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.641731    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:32.713126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.712596    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbntp\" (UniqueName: \"kubernetes.io/projected/79337cdb-cbd2-441f-9011-e78a674f53a8-kube-api-access-tbntp\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:32.713126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.712637    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79337cdb-cbd2-441f-9011-e78a674f53a8-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:32.787893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:32.787852    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:33.119985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.119904    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:33.170299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.170273    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"
Oct 30 06:22:33.170485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.170276    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5" event={"ID":"79337cdb-cbd2-441f-9011-e78a674f53a8","Type":"ContainerDied","Data":"444bf52a2a4422eb7d34faad4e73e66eab1c4f8a3df5a2a72060a6db25187a22"}
Oct 30 06:22:33.170485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.170403    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="444bf52a2a4422eb7d34faad4e73e66eab1c4f8a3df5a2a72060a6db25187a22"
Oct 30 06:22:33.282512 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.282485    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4899d4a4-27cc-4efa-a9ea-0017342bb0fc" path="/var/lib/kubelet/pods/4899d4a4-27cc-4efa-a9ea-0017342bb0fc/volumes"
Oct 30 06:22:33.535144 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.535120    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:22:33.725576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725494    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istiod-ca-cert\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.725720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725638    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-kube-api-access-x75ns\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-kube-api-access-x75ns\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.725720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725658    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-credential-socket\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.725720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725695    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-token\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-token\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.725848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725728    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-certs\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.725848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725761    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-podinfo\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.725848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725833    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-data\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.726021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725888    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-socket\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.726021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725930    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-envoy\") pod \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\" (UID: \"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6\") "
Oct 30 06:22:33.726021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.725985    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-credential-socket" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:33.726206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726019    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istiod-ca-cert" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:33.726206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726099    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-certs" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:33.726206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726116    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-data" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:33.726468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726445    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-socket" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:33.726840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726815    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:33.726840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726843    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:33.727037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726857    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:33.727037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726872    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:33.727037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.726884    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:33.728326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.728299    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-token" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:33.728436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.728355    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-kube-api-access-x75ns" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "kube-api-access-x75ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:33.728436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.728411    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-envoy" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:33.728570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.728550    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-podinfo" pod "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" (UID: "cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:22:33.828723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.828686    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:33.828723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.828718    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75ns\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-kube-api-access-x75ns\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:33.828723 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.828730    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:33.828957 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:33.828739    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:34.178521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.178488    5250 generic.go:356] "Generic (PLEG): container finished" podID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerID="d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da" exitCode=0
Oct 30 06:22:34.178953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.178542    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"
Oct 30 06:22:34.178953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.178569    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" event={"ID":"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6","Type":"ContainerDied","Data":"d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da"}
Oct 30 06:22:34.178953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.178625    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q" event={"ID":"cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6","Type":"ContainerDied","Data":"27ee0653650b4cb5fbe2395a124688006dbdd5449ece831226f02fa57e7a1056"}
Oct 30 06:22:34.178953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.178647    5250 scope.go:122] "RemoveContainer" containerID="d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da"
Oct 30 06:22:34.190287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.190270    5250 scope.go:122] "RemoveContainer" containerID="d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da"
Oct 30 06:22:34.190584 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:22:34.190566    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da\": container with ID starting with d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da not found: ID does not exist" containerID="d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da"
Oct 30 06:22:34.190635 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.190592    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da"} err="failed to get container status \"d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da\": rpc error: code = NotFound desc = could not find container \"d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da\": container with ID starting with d7deebd853e20530cd7fc234b518de7bd8cdd0cd39c69979fc1961ba789333da not found: ID does not exist"
Oct 30 06:22:34.234676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.234656    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"]
Oct 30 06:22:34.246459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.246437    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-pjh5q"]
Oct 30 06:22:34.484671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.484583    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:34.626205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:34.626146    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:35.282499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:35.282467    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" path="/var/lib/kubelet/pods/cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6/volumes"
Oct 30 06:22:35.435971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:35.435935    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:37.083055 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:37.082970    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:37.351321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:37.351239    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:39.043373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:39.043334    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:39.291205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:39.291160    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:39.448624 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:39.448590    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:39.719295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:39.719192    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:40.909472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:40.909433    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:42.664321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:42.664291    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:44.341029 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:44.340993    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:44.801793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:44.801753    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"]
Oct 30 06:22:44.801974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:44.801953    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy" containerID="cri-o://af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff" gracePeriod=30
Oct 30 06:22:45.893103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:45.893065    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:47.178920 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.178885    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:47.350412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.350380    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:47.490359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.490269    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"]
Oct 30 06:22:47.490854 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.490811    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn" podUID="57f4efea-a882-4792-a1ca-2c4b5ebcd44d" containerName="maas-api" containerID="cri-o://abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c" gracePeriod=30
Oct 30 06:22:47.502246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.502217    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:47.651063 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.651010    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:47.786079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.786053    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 06:22:47.794930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.794901    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:47.920680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.911076    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-openshift-service-ca\") pod \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") "
Oct 30 06:22:47.920680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.911156    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-maas-api-tls\") pod \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") "
Oct 30 06:22:47.920680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.911232    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-kube-api-access-2fwsz\" (UniqueName: \"kubernetes.io/projected/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-kube-api-access-2fwsz\") pod \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") "
Oct 30 06:22:47.920680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.911400    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-metrics-certs\") pod \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\" (UID: \"57f4efea-a882-4792-a1ca-2c4b5ebcd44d\") "
Oct 30 06:22:47.924458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.924424    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-openshift-service-ca" pod "57f4efea-a882-4792-a1ca-2c4b5ebcd44d" (UID: "57f4efea-a882-4792-a1ca-2c4b5ebcd44d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:47.954286 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.954232    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-kube-api-access-2fwsz" pod "57f4efea-a882-4792-a1ca-2c4b5ebcd44d" (UID: "57f4efea-a882-4792-a1ca-2c4b5ebcd44d"). InnerVolumeSpecName "kube-api-access-2fwsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:47.954710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.954678    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-metrics-certs" pod "57f4efea-a882-4792-a1ca-2c4b5ebcd44d" (UID: "57f4efea-a882-4792-a1ca-2c4b5ebcd44d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:22:47.965889 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:47.964575    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-maas-api-tls" pod "57f4efea-a882-4792-a1ca-2c4b5ebcd44d" (UID: "57f4efea-a882-4792-a1ca-2c4b5ebcd44d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:22:48.013397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.013313    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:48.013397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.013342    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:48.013397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.013353    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:48.013397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.013361    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwsz\" (UniqueName: \"kubernetes.io/projected/57f4efea-a882-4792-a1ca-2c4b5ebcd44d-kube-api-access-2fwsz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:48.239842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.239807    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:48.297470 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.297385    5250 generic.go:356] "Generic (PLEG): container finished" podID="57f4efea-a882-4792-a1ca-2c4b5ebcd44d" containerID="abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c" exitCode=0
Oct 30 06:22:48.297470 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.297423    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"
Oct 30 06:22:48.297470 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.297446    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn" event={"ID":"57f4efea-a882-4792-a1ca-2c4b5ebcd44d","Type":"ContainerDied","Data":"abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c"}
Oct 30 06:22:48.297732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.297490    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn" event={"ID":"57f4efea-a882-4792-a1ca-2c4b5ebcd44d","Type":"ContainerDied","Data":"30be399f4af84fdab693df1eefd4ecc7015143a4e0d2c31e93b949b5a6f8b9cf"}
Oct 30 06:22:48.297732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.297511    5250 scope.go:122] "RemoveContainer" containerID="abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c"
Oct 30 06:22:48.309209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.309157    5250 scope.go:122] "RemoveContainer" containerID="abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c"
Oct 30 06:22:48.309473 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:22:48.309452    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c\": container with ID starting with abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c not found: ID does not exist" containerID="abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c"
Oct 30 06:22:48.309532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.309482    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c"} err="failed to get container status \"abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c\": rpc error: code = NotFound desc = could not find container \"abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c\": container with ID starting with abd6afe0564c1ac40d3e82b53a5e8d0e058de25adb0b43ad5946fe32cd1d6a6c not found: ID does not exist"
Oct 30 06:22:48.383035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.383007    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"]
Oct 30 06:22:48.396730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.396702    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a14091-6b445d5f68-hnfhn"]
Oct 30 06:22:48.492885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.492850    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:48.633297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.633270    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:48.878314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:48.878283    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:49.281193 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:49.281152    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f4efea-a882-4792-a1ca-2c4b5ebcd44d" path="/var/lib/kubelet/pods/57f4efea-a882-4792-a1ca-2c4b5ebcd44d/volumes"
Oct 30 06:22:50.071257 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.071171    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:22:50.138861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.138829    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-podinfo\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.138942    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-certs\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139019    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-socket\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139051    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-data\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139118    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-credential-socket\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139154    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-kube-api-access-vsq7n\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-kube-api-access-vsq7n\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139274    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-token\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-token\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139279    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-certs" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:50.139403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139370    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-envoy\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139411    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istiod-ca-cert\") pod \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\" (UID: \"072333b6-e21e-4adf-ab3c-c53e5169a9ea\") "
Oct 30 06:22:50.139586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139444    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-data" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:50.139586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.139574    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-socket" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:50.140128 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.140103    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.140361 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.140328    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.140501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.140486    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.140778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.140744    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istiod-ca-cert" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:50.140950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.140926    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-credential-socket" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:50.141511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.141489    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-podinfo" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:22:50.143074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.143045    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-token" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:50.143536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.143516    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-envoy" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:50.144371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.144355    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-kube-api-access-vsq7n" pod "072333b6-e21e-4adf-ab3c-c53e5169a9ea" (UID: "072333b6-e21e-4adf-ab3c-c53e5169a9ea"). InnerVolumeSpecName "kube-api-access-vsq7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:50.158171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.158147    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:50.249743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.244392    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.249743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.244422    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.249743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.244435    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.249743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.244449    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072333b6-e21e-4adf-ab3c-c53e5169a9ea-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.249743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.244462    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072333b6-e21e-4adf-ab3c-c53e5169a9ea-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.249743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.244479    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsq7n\" (UniqueName: \"kubernetes.io/projected/072333b6-e21e-4adf-ab3c-c53e5169a9ea-kube-api-access-vsq7n\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:50.314897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.314862    5250 generic.go:356] "Generic (PLEG): container finished" podID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerID="af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff" exitCode=0
Oct 30 06:22:50.315399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.314909    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"
Oct 30 06:22:50.315399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.314937    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" event={"ID":"072333b6-e21e-4adf-ab3c-c53e5169a9ea","Type":"ContainerDied","Data":"af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff"}
Oct 30 06:22:50.315399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.314991    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4" event={"ID":"072333b6-e21e-4adf-ab3c-c53e5169a9ea","Type":"ContainerDied","Data":"3f1a53b9a89ae45eafecbf15953cb7fb78df7676a2615a5faa0218bd476f2cc1"}
Oct 30 06:22:50.315399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.315014    5250 scope.go:122] "RemoveContainer" containerID="af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff"
Oct 30 06:22:50.326441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.326423    5250 scope.go:122] "RemoveContainer" containerID="af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff"
Oct 30 06:22:50.326730 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:22:50.326706    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff\": container with ID starting with af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff not found: ID does not exist" containerID="af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff"
Oct 30 06:22:50.326797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.326743    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff"} err="failed to get container status \"af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff\": rpc error: code = NotFound desc = could not find container \"af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff\": container with ID starting with af50a88ed0026bd1d5d1cfb8cbb89557a334a59759585905703631680f1fbaff not found: ID does not exist"
Oct 30 06:22:50.367800 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.367773    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"]
Oct 30 06:22:50.372767 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.372728    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-2wrn4"]
Oct 30 06:22:50.429200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.429114    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"]
Oct 30 06:22:50.565993 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.565948    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:50.704940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:50.704879    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:51.214479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:51.210605    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"]
Oct 30 06:22:51.214479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:51.210939    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy" containerID="cri-o://9dc038e17ec366f7867b7486d98781ddf58731eb4714706c10aca05328163b53" gracePeriod=30
Oct 30 06:22:51.281704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:51.281671    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" path="/var/lib/kubelet/pods/072333b6-e21e-4adf-ab3c-c53e5169a9ea/volumes"
Oct 30 06:22:51.635353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:51.635318    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:52.359185 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:52.359122    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:52.495543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:52.495509    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"]
Oct 30 06:22:52.506702 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:52.506670    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b8f87-fvcv5"]
Oct 30 06:22:52.506977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:52.506958    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:52.653302 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:52.653271    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:52.996636 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:52.996558    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:53.281498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:53.281413    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79337cdb-cbd2-441f-9011-e78a674f53a8" path="/var/lib/kubelet/pods/79337cdb-cbd2-441f-9011-e78a674f53a8/volumes"
Oct 30 06:22:54.190306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:54.190269    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:55.142164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.142118    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:55.286388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.286358    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:55.424054 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.423968    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"]
Oct 30 06:22:55.424304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.424269    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy" containerID="cri-o://63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7" gracePeriod=30
Oct 30 06:22:55.429842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.429813    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" podUID="ac2a746e-35d3-408d-b9c1-66890f8ad655" containerName="payload-processing" containerID="cri-o://1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc" gracePeriod=25
Oct 30 06:22:55.474155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.474127    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:55.627744 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.627694    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" podUID="ac2a746e-35d3-408d-b9c1-66890f8ad655" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:22:55.768506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.767542    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"]
Oct 30 06:22:55.769146 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.768990    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy" containerID="cri-o://52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94" gracePeriod=30
Oct 30 06:22:55.775733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.775699    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:55.945482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.945447    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:55.947380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.947358    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"]
Oct 30 06:22:55.947937 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.947924    5250 cpu_manager.go:414] "containerMap: removing container" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy"
Oct 30 06:22:55.947980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.947940    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy"
Oct 30 06:22:55.947980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.947971    5250 cpu_manager.go:414] "containerMap: removing container" podUID="57f4efea-a882-4792-a1ca-2c4b5ebcd44d" containerName="maas-api"
Oct 30 06:22:55.947980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.947976    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f4efea-a882-4792-a1ca-2c4b5ebcd44d" containerName="maas-api"
Oct 30 06:22:55.948078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.947986    5250 cpu_manager.go:414] "containerMap: removing container" podUID="79337cdb-cbd2-441f-9011-e78a674f53a8" containerName="revoke-keys"
Oct 30 06:22:55.948078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.947992    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79337cdb-cbd2-441f-9011-e78a674f53a8" containerName="revoke-keys"
Oct 30 06:22:55.948078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.948002    5250 cpu_manager.go:414] "containerMap: removing container" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy"
Oct 30 06:22:55.948078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.948006    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy"
Oct 30 06:22:55.948231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.948084    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f4efea-a882-4792-a1ca-2c4b5ebcd44d" containerName="maas-api"
Oct 30 06:22:55.948231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.948094    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="79337cdb-cbd2-441f-9011-e78a674f53a8" containerName="revoke-keys"
Oct 30 06:22:55.948231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.948101    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="072333b6-e21e-4adf-ab3c-c53e5169a9ea" containerName="istio-proxy"
Oct 30 06:22:55.948231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.948110    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf03b07e-8e6b-4b74-aa6f-2c59bc4ff5d6" containerName="istio-proxy"
Oct 30 06:22:55.951811 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.951786    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:55.966295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.966271    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:22:55.966588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.966564    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:22:55.971228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:55.971205    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"]
Oct 30 06:22:56.069100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.069000    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"]
Oct 30 06:22:56.069298 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.069278    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy" containerID="cri-o://0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9" gracePeriod=30
Oct 30 06:22:56.134366 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.134329    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:56.146537 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.146507    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhjt\" (UniqueName: \"kubernetes.io/projected/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-kube-api-access-9hhjt\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.146700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.146678    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.146773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.146735    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.248149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.248118    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhjt\" (UniqueName: \"kubernetes.io/projected/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-kube-api-access-9hhjt\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.248298 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.248278    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.248379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.248330    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.249032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.248973    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.250747 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.250723    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.270823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.270803    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhjt\" (UniqueName: \"kubernetes.io/projected/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-kube-api-access-9hhjt\") pod \"maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.360770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.360678    5250 generic.go:356] "Generic (PLEG): container finished" podID="e62d5387-cd7e-4673-8cd9-e05738171903" containerID="9dc038e17ec366f7867b7486d98781ddf58731eb4714706c10aca05328163b53" exitCode=0
Oct 30 06:22:56.360770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.360756    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" event={"ID":"e62d5387-cd7e-4673-8cd9-e05738171903","Type":"ContainerDied","Data":"9dc038e17ec366f7867b7486d98781ddf58731eb4714706c10aca05328163b53"}
Oct 30 06:22:56.566204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.566147    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:56.867379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.867357    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:22:56.877612 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:56.877588    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"]
Oct 30 06:22:56.878381 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:22:56.878355    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77a6f10_83c6_4b28_bc30_c7f9ca5cb846.slice/crio-637be0d3725a4587575132fcf8273f6d14d7f6d784e120a41569dd442b82994b WatchSource:0}: Error finding container 637be0d3725a4587575132fcf8273f6d14d7f6d784e120a41569dd442b82994b: Status 404 returned error can't find the container with id 637be0d3725a4587575132fcf8273f6d14d7f6d784e120a41569dd442b82994b
Oct 30 06:22:57.056648 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056612    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e62d5387-cd7e-4673-8cd9-e05738171903-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62d5387-cd7e-4673-8cd9-e05738171903-istio-podinfo\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.056648 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056653    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-credential-socket\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.056861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056758    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-certs\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.056861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056821    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-istio-token\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-istio-token\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.056861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056853    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-socket\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.057032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056921    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-data\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.057032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056956    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62d5387-cd7e-4673-8cd9-e05738171903-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62d5387-cd7e-4673-8cd9-e05738171903-istiod-ca-cert\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.057032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056984    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-envoy\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.057032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.056990    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-credential-socket" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:57.057269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.057038    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-certs" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:57.057269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.057061    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-kube-api-access-6gpfq\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-kube-api-access-6gpfq\") pod \"e62d5387-cd7e-4673-8cd9-e05738171903\" (UID: \"e62d5387-cd7e-4673-8cd9-e05738171903\") "
Oct 30 06:22:57.057400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.057314    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-data" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:57.057469 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.057389    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62d5387-cd7e-4673-8cd9-e05738171903-istiod-ca-cert" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:57.057692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.057508    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-socket" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:57.058073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.058046    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.058209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.058079    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.058209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.058095    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.058209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.058110    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.058209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.058124    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62d5387-cd7e-4673-8cd9-e05738171903-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.059119 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.059099    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e62d5387-cd7e-4673-8cd9-e05738171903-istio-podinfo" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:22:57.059381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.059364    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-istio-token" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:57.059451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.059417    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-kube-api-access-6gpfq" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "kube-api-access-6gpfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:57.059528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.059514    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-envoy" pod "e62d5387-cd7e-4673-8cd9-e05738171903" (UID: "e62d5387-cd7e-4673-8cd9-e05738171903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:22:57.159056 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.159015    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.159056 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.159047    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62d5387-cd7e-4673-8cd9-e05738171903-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.159316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.159061    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gpfq\" (UniqueName: \"kubernetes.io/projected/e62d5387-cd7e-4673-8cd9-e05738171903-kube-api-access-6gpfq\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.159316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.159073    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62d5387-cd7e-4673-8cd9-e05738171903-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:57.368870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.368829    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj" event={"ID":"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846","Type":"ContainerStarted","Data":"ea401194053d56af789e5d9baf1652c7d2580aa7c343abf2681e42bc802a4341"}
Oct 30 06:22:57.369334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.368891    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj" event={"ID":"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846","Type":"ContainerStarted","Data":"637be0d3725a4587575132fcf8273f6d14d7f6d784e120a41569dd442b82994b"}
Oct 30 06:22:57.370377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.370352    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f" event={"ID":"e62d5387-cd7e-4673-8cd9-e05738171903","Type":"ContainerDied","Data":"c8b11ab5f052e6add459c682ec4e993d44dc08d66226b9292f1eb6404cbb48e4"}
Oct 30 06:22:57.370490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.370380    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"
Oct 30 06:22:57.370490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.370402    5250 scope.go:122] "RemoveContainer" containerID="9dc038e17ec366f7867b7486d98781ddf58731eb4714706c10aca05328163b53"
Oct 30 06:22:57.393086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.393061    5250 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:22:57.393200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.393118    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:22:57.411511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.411431    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj" podStartSLOduration=2.411418527 podStartE2EDuration="2.411418527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:57.409050046 +0000 UTC m=+3092.240087264" watchObservedRunningTime="2026-10-30 06:22:57.411418527 +0000 UTC m=+3092.242455743"
Oct 30 06:22:57.448279 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.448258    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"]
Oct 30 06:22:57.460311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.460278    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-hvt5f"]
Oct 30 06:22:57.827500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:57.827456    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:58.378504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:58.378476    5250 generic.go:356] "Generic (PLEG): container finished" podID="c77a6f10-83c6-4b28-bc30-c7f9ca5cb846" containerID="ea401194053d56af789e5d9baf1652c7d2580aa7c343abf2681e42bc802a4341" exitCode=0
Oct 30 06:22:58.378927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:58.378554    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj" event={"ID":"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846","Type":"ContainerDied","Data":"ea401194053d56af789e5d9baf1652c7d2580aa7c343abf2681e42bc802a4341"}
Oct 30 06:22:59.242947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.242910    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:59.282353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.282312    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" path="/var/lib/kubelet/pods/e62d5387-cd7e-4673-8cd9-e05738171903/volumes"
Oct 30 06:22:59.382980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.382946    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:59.569211 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.569167    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:22:59.573280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.573254    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:22:59.584810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.584789    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-xks-ca\") pod \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") "
Oct 30 06:22:59.584930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.584826    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-kube-api-access-9hhjt\" (UniqueName: \"kubernetes.io/projected/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-kube-api-access-9hhjt\") pod \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") "
Oct 30 06:22:59.584930 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.584854    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-openshift-ca\") pod \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\" (UID: \"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846\") "
Oct 30 06:22:59.585282 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.585251    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-openshift-ca" pod "c77a6f10-83c6-4b28-bc30-c7f9ca5cb846" (UID: "c77a6f10-83c6-4b28-bc30-c7f9ca5cb846"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:22:59.585440 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.585421    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:59.587561 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.587538    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-xks-ca" pod "c77a6f10-83c6-4b28-bc30-c7f9ca5cb846" (UID: "c77a6f10-83c6-4b28-bc30-c7f9ca5cb846"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:22:59.587678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.587657    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-kube-api-access-9hhjt" pod "c77a6f10-83c6-4b28-bc30-c7f9ca5cb846" (UID: "c77a6f10-83c6-4b28-bc30-c7f9ca5cb846"). InnerVolumeSpecName "kube-api-access-9hhjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:22:59.686878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.686838    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:59.686878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.686882    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hhjt\" (UniqueName: \"kubernetes.io/projected/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846-kube-api-access-9hhjt\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:22:59.983672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:22:59.983636    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:00.123090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.123032    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:00.173128 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.173098    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"]
Oct 30 06:23:00.173671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.173657    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy"
Oct 30 06:23:00.173726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.173673    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy"
Oct 30 06:23:00.173726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.173708    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c77a6f10-83c6-4b28-bc30-c7f9ca5cb846" containerName="revoke-keys"
Oct 30 06:23:00.173726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.173713    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c77a6f10-83c6-4b28-bc30-c7f9ca5cb846" containerName="revoke-keys"
Oct 30 06:23:00.173825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.173789    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c77a6f10-83c6-4b28-bc30-c7f9ca5cb846" containerName="revoke-keys"
Oct 30 06:23:00.173825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.173802    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62d5387-cd7e-4673-8cd9-e05738171903" containerName="istio-proxy"
Oct 30 06:23:00.177204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.177190    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.192420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.192391    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.192604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.192580    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbft\" (UniqueName: \"kubernetes.io/projected/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-kube-api-access-lgbft\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.192697 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.192666    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.196113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.196086    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"]
Oct 30 06:23:00.293527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.293450    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.293674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.293561    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbft\" (UniqueName: \"kubernetes.io/projected/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-kube-api-access-lgbft\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.293674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.293605    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.294233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.294209    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.296251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.296228    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.316337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.316316    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbft\" (UniqueName: \"kubernetes.io/projected/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-kube-api-access-lgbft\") pod \"maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.395931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.395905    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"
Oct 30 06:23:00.395931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.395923    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj" event={"ID":"c77a6f10-83c6-4b28-bc30-c7f9ca5cb846","Type":"ContainerDied","Data":"637be0d3725a4587575132fcf8273f6d14d7f6d784e120a41569dd442b82994b"}
Oct 30 06:23:00.396374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.395962    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637be0d3725a4587575132fcf8273f6d14d7f6d784e120a41569dd442b82994b"
Oct 30 06:23:00.491327 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.491300    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:00.716870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.716847    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:23:00.800684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.800608    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-socket\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.800684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.800675    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-certs\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.801237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801155    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-socket" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:00.801237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801227    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-certs" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:00.801379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801296    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-envoy\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.801524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801503    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-data\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.801592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801543    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-istio-token\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-istio-token\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.801592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801572    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-credential-socket\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.801733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801616    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b79e875-5537-492e-aa94-3a015701a0f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b79e875-5537-492e-aa94-3a015701a0f8-istio-podinfo\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.801733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801652    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b79e875-5537-492e-aa94-3a015701a0f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b79e875-5537-492e-aa94-3a015701a0f8-istiod-ca-cert\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.801733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801694    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-kube-api-access-fkq47\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-kube-api-access-fkq47\") pod \"3b79e875-5537-492e-aa94-3a015701a0f8\" (UID: \"3b79e875-5537-492e-aa94-3a015701a0f8\") "
Oct 30 06:23:00.802067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801802    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-data" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:00.802067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.801971    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-credential-socket" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:00.802273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.802253    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:00.802395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.802277    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:00.802395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.802291    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:00.802395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.802306    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:00.803114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.803090    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b79e875-5537-492e-aa94-3a015701a0f8-istiod-ca-cert" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:00.804208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.804155    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-envoy" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:00.804898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.804876    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b79e875-5537-492e-aa94-3a015701a0f8-istio-podinfo" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:23:00.805613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.805595    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-istio-token" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:00.807728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.807704    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"]
Oct 30 06:23:00.809056 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.809038    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-kube-api-access-fkq47" pod "3b79e875-5537-492e-aa94-3a015701a0f8" (UID: "3b79e875-5537-492e-aa94-3a015701a0f8"). InnerVolumeSpecName "kube-api-access-fkq47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:00.813632 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:23:00.813610    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f215d4a_d20f_4d69_9606_7e0980b3c4a2.slice/crio-8d0cfcf435f7db9d0cbdaa79b34429400f46c27459ba0d5716cd06bf4755cf97 WatchSource:0}: Error finding container 8d0cfcf435f7db9d0cbdaa79b34429400f46c27459ba0d5716cd06bf4755cf97: Status 404 returned error can't find the container with id 8d0cfcf435f7db9d0cbdaa79b34429400f46c27459ba0d5716cd06bf4755cf97
Oct 30 06:23:00.902962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.902934    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:00.902962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.902964    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b79e875-5537-492e-aa94-3a015701a0f8-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:00.903110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.902974    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b79e875-5537-492e-aa94-3a015701a0f8-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:00.903110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.902983    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkq47\" (UniqueName: \"kubernetes.io/projected/3b79e875-5537-492e-aa94-3a015701a0f8-kube-api-access-fkq47\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:00.903110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:00.902992    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b79e875-5537-492e-aa94-3a015701a0f8-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.065500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.065474    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:23:01.104913 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.104887    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-envoy\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.104942    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-socket\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105014    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istiod-ca-cert\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105047    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-data\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105122    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-kube-api-access-stcc4\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-kube-api-access-stcc4\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105258    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-certs\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105286    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-socket" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.105431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105321    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-podinfo\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105402    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-credential-socket\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105539 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105434    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-token\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-token\") pod \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\" (UID: \"e6bec144-e4ab-4209-98a2-019bdfed2fe6\") "
Oct 30 06:23:01.105597 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105563    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-certs" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.105654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105591    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-data" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.105654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105533    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istiod-ca-cert" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:01.105879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105852    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-credential-socket" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.106006 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.105994    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.106077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.106007    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.106077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.106017    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.106077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.106035    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.106077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.106051    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.107374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.107348    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-envoy" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.107719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.107692    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-kube-api-access-stcc4" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "kube-api-access-stcc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:01.107829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.107718    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-podinfo" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:23:01.107916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.107900    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-token" pod "e6bec144-e4ab-4209-98a2-019bdfed2fe6" (UID: "e6bec144-e4ab-4209-98a2-019bdfed2fe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:01.207116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.207083    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stcc4\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-kube-api-access-stcc4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.207116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.207117    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.207330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.207132    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.207330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.207143    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6bec144-e4ab-4209-98a2-019bdfed2fe6-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.326926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.326902    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 06:23:01.407751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.407713    5250 generic.go:356] "Generic (PLEG): container finished" podID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerID="0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9" exitCode=0
Oct 30 06:23:01.408296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.407789    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"
Oct 30 06:23:01.408296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.407799    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" event={"ID":"8ae41e05-46f8-418b-ab84-968a0e5a23b1","Type":"ContainerDied","Data":"0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9"}
Oct 30 06:23:01.408296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.407869    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx" event={"ID":"8ae41e05-46f8-418b-ab84-968a0e5a23b1","Type":"ContainerDied","Data":"ce6664de4234c1db6193c3da81309b11220850b81bba90b1ae5a7581aad686c2"}
Oct 30 06:23:01.408296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.407902    5250 scope.go:122] "RemoveContainer" containerID="0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9"
Oct 30 06:23:01.409196 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.409157    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-socket\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.409353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.409251    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-data\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.409518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.409461    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-socket" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.409605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.409532    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-data" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.410253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.410230    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-token\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-token\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.411299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.411277    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-kube-api-access-6nlp9\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-kube-api-access-6nlp9\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.411506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.411487    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-credential-socket\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.411651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.411636    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-certs\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.411796 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.411781    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-envoy\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.411922 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.411907    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-podinfo\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.412045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.412029    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istiod-ca-cert\") pod \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\" (UID: \"8ae41e05-46f8-418b-ab84-968a0e5a23b1\") "
Oct 30 06:23:01.413148 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.413126    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.413368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.413352    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.413485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.412055    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r" event={"ID":"7f215d4a-d20f-4d69-9606-7e0980b3c4a2","Type":"ContainerStarted","Data":"b3ffa02275d9d804b1f20a91c4d8bd30e9e8d157ed6ca344b83594499cc6043a"}
Oct 30 06:23:01.413620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.413603    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r" event={"ID":"7f215d4a-d20f-4d69-9606-7e0980b3c4a2","Type":"ContainerStarted","Data":"8d0cfcf435f7db9d0cbdaa79b34429400f46c27459ba0d5716cd06bf4755cf97"}
Oct 30 06:23:01.413751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.412327    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-certs" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.413868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.412553    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-credential-socket" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.414037 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.413625    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istiod-ca-cert" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:01.414246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.413743    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-token" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:01.414899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.414806    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-kube-api-access-6nlp9" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "kube-api-access-6nlp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:01.415566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.415540    5250 generic.go:356] "Generic (PLEG): container finished" podID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerID="52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94" exitCode=0
Oct 30 06:23:01.415683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.415673    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" event={"ID":"e6bec144-e4ab-4209-98a2-019bdfed2fe6","Type":"ContainerDied","Data":"52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94"}
Oct 30 06:23:01.415743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.415709    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d" event={"ID":"e6bec144-e4ab-4209-98a2-019bdfed2fe6","Type":"ContainerDied","Data":"192a5b8d8e840c7a1d1c1c2d06c24bf4089456bd1680bf57337dfaa368a58b45"}
Oct 30 06:23:01.416433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.416357    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-envoy" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:01.416555 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.416501    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-podinfo" pod "8ae41e05-46f8-418b-ab84-968a0e5a23b1" (UID: "8ae41e05-46f8-418b-ab84-968a0e5a23b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:23:01.417338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.417299    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"
Oct 30 06:23:01.417973 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.417954    5250 generic.go:356] "Generic (PLEG): container finished" podID="3b79e875-5537-492e-aa94-3a015701a0f8" containerID="63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7" exitCode=0
Oct 30 06:23:01.418073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.418012    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" event={"ID":"3b79e875-5537-492e-aa94-3a015701a0f8","Type":"ContainerDied","Data":"63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7"}
Oct 30 06:23:01.418073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.418005    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"
Oct 30 06:23:01.418073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.418050    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf" event={"ID":"3b79e875-5537-492e-aa94-3a015701a0f8","Type":"ContainerDied","Data":"682d00cbe2f9d5b7807772f869732d9aac00052485e5e44fff24f777779ba569"}
Oct 30 06:23:01.423642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.423620    5250 scope.go:122] "RemoveContainer" containerID="0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e"
Oct 30 06:23:01.435263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.435240    5250 scope.go:122] "RemoveContainer" containerID="0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9"
Oct 30 06:23:01.435581 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:01.435558    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9\": container with ID starting with 0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9 not found: ID does not exist" containerID="0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9"
Oct 30 06:23:01.435661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.435586    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9"} err="failed to get container status \"0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9\": rpc error: code = NotFound desc = could not find container \"0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9\": container with ID starting with 0d8c838f551ae37ced8127725c1ed56af5e3f99e39128b46f8dce1349306a3c9 not found: ID does not exist"
Oct 30 06:23:01.435661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.435603    5250 scope.go:122] "RemoveContainer" containerID="0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e"
Oct 30 06:23:01.435856 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:01.435841    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e\": container with ID starting with 0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e not found: ID does not exist" containerID="0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e"
Oct 30 06:23:01.435918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.435868    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e"} err="failed to get container status \"0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e\": rpc error: code = NotFound desc = could not find container \"0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e\": container with ID starting with 0f27937a0875a89b8e871bbd46721b288c72bfd04b3c4f6e95092f6498177a2e not found: ID does not exist"
Oct 30 06:23:01.435918 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.435881    5250 scope.go:122] "RemoveContainer" containerID="52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94"
Oct 30 06:23:01.447796 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.447768    5250 scope.go:122] "RemoveContainer" containerID="52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94"
Oct 30 06:23:01.448060 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:01.448035    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94\": container with ID starting with 52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94 not found: ID does not exist" containerID="52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94"
Oct 30 06:23:01.448159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.448069    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94"} err="failed to get container status \"52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94\": rpc error: code = NotFound desc = could not find container \"52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94\": container with ID starting with 52fb954929a35b037c232bd81408a1b5a7f81ebfb7f0d63514b60a1f484d7d94 not found: ID does not exist"
Oct 30 06:23:01.448159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.448089    5250 scope.go:122] "RemoveContainer" containerID="63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7"
Oct 30 06:23:01.452781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.452760    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:01.459404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.459385    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"]
Oct 30 06:23:01.465344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.464674    5250 scope.go:122] "RemoveContainer" containerID="304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62"
Oct 30 06:23:01.469825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.469802    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7fdb95-openshift-default-54b65bf649-8zdsf"]
Oct 30 06:23:01.477735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.477715    5250 scope.go:122] "RemoveContainer" containerID="63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7"
Oct 30 06:23:01.478103 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:01.478078    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7\": container with ID starting with 63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7 not found: ID does not exist" containerID="63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7"
Oct 30 06:23:01.478197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.478118    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7"} err="failed to get container status \"63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7\": rpc error: code = NotFound desc = could not find container \"63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7\": container with ID starting with 63abb789dc2120d81c0362c44c73f3c7087267e22a7a1c5e361ebf6ea3d3cda7 not found: ID does not exist"
Oct 30 06:23:01.478197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.478146    5250 scope.go:122] "RemoveContainer" containerID="304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62"
Oct 30 06:23:01.478505 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:01.478482    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62\": container with ID starting with 304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62 not found: ID does not exist" containerID="304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62"
Oct 30 06:23:01.478578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.478513    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62"} err="failed to get container status \"304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62\": rpc error: code = NotFound desc = could not find container \"304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62\": container with ID starting with 304b4b58a8009a4c726741ce63a3a99f54ff4c61b775a01ee7e0ea7ad3d8fb62 not found: ID does not exist"
Oct 30 06:23:01.507927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.507873    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r" podStartSLOduration=1.507856093 podStartE2EDuration="1.507856093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:01.503543187 +0000 UTC m=+3096.334580406" watchObservedRunningTime="2026-10-30 06:23:01.507856093 +0000 UTC m=+3096.338893311"
Oct 30 06:23:01.514848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.514821    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.514848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.514843    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.514848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.514851    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.515081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.514859    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nlp9\" (UniqueName: \"kubernetes.io/projected/8ae41e05-46f8-418b-ab84-968a0e5a23b1-kube-api-access-6nlp9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.515081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.514869    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.515081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.514878    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.515081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.514887    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae41e05-46f8-418b-ab84-968a0e5a23b1-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:01.544933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.543731    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"]
Oct 30 06:23:01.552808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.548714    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-kdc2d"]
Oct 30 06:23:01.750870 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.750842    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:01.756334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.756311    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"]
Oct 30 06:23:01.762281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.762258    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-lj2gx"]
Oct 30 06:23:01.896819 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:01.896775    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:02.136813 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:02.136779    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:02.283966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:02.280906    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:02.374453 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:02.374414    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:23:02.427473 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:02.427441    5250 generic.go:356] "Generic (PLEG): container finished" podID="7f215d4a-d20f-4d69-9606-7e0980b3c4a2" containerID="b3ffa02275d9d804b1f20a91c4d8bd30e9e8d157ed6ca344b83594499cc6043a" exitCode=0
Oct 30 06:23:02.427929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:02.427532    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r" event={"ID":"7f215d4a-d20f-4d69-9606-7e0980b3c4a2","Type":"ContainerDied","Data":"b3ffa02275d9d804b1f20a91c4d8bd30e9e8d157ed6ca344b83594499cc6043a"}
Oct 30 06:23:02.766977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:02.766943    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:03.280746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.280714    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" path="/var/lib/kubelet/pods/3b79e875-5537-492e-aa94-3a015701a0f8/volumes"
Oct 30 06:23:03.281193 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.281166    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" path="/var/lib/kubelet/pods/8ae41e05-46f8-418b-ab84-968a0e5a23b1/volumes"
Oct 30 06:23:03.281613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.281600    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" path="/var/lib/kubelet/pods/e6bec144-e4ab-4209-98a2-019bdfed2fe6/volumes"
Oct 30 06:23:03.590518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.590490    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:03.638829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.638798    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-openshift-ca\") pod \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") "
Oct 30 06:23:03.639021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.638865    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-kube-api-access-lgbft\" (UniqueName: \"kubernetes.io/projected/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-kube-api-access-lgbft\") pod \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") "
Oct 30 06:23:03.639021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.638923    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-xks-ca\") pod \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\" (UID: \"7f215d4a-d20f-4d69-9606-7e0980b3c4a2\") "
Oct 30 06:23:03.639293 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.639271    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-openshift-ca" pod "7f215d4a-d20f-4d69-9606-7e0980b3c4a2" (UID: "7f215d4a-d20f-4d69-9606-7e0980b3c4a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:03.639533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.639508    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:03.641450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.641421    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-xks-ca" pod "7f215d4a-d20f-4d69-9606-7e0980b3c4a2" (UID: "7f215d4a-d20f-4d69-9606-7e0980b3c4a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:23:03.641747 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.641723    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-kube-api-access-lgbft" pod "7f215d4a-d20f-4d69-9606-7e0980b3c4a2" (UID: "7f215d4a-d20f-4d69-9606-7e0980b3c4a2"). InnerVolumeSpecName "kube-api-access-lgbft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:03.740758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.740724    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgbft\" (UniqueName: \"kubernetes.io/projected/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-kube-api-access-lgbft\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:03.740758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:03.740755    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f215d4a-d20f-4d69-9606-7e0980b3c4a2-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:04.150134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:04.150081    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:04.444753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:04.444676    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r" event={"ID":"7f215d4a-d20f-4d69-9606-7e0980b3c4a2","Type":"ContainerDied","Data":"8d0cfcf435f7db9d0cbdaa79b34429400f46c27459ba0d5716cd06bf4755cf97"}
Oct 30 06:23:04.444753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:04.444720    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"
Oct 30 06:23:04.444951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:04.444724    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d0cfcf435f7db9d0cbdaa79b34429400f46c27459ba0d5716cd06bf4755cf97"
Oct 30 06:23:04.999390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:04.999360    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"]
Oct 30 06:23:05.131413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:05.131378    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:05.627446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:05.627403    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" podUID="ac2a746e-35d3-408d-b9c1-66890f8ad655" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:05.662758 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:05.662715    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:07.297191 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:07.297136    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:08.842811 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:08.842773    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:08.989323 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:08.989288    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:09.126635 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:09.126544    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"]
Oct 30 06:23:09.126887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:09.126856    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy" containerID="cri-o://efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750" gracePeriod=30
Oct 30 06:23:09.182263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:09.182210    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:10.517617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:10.517583    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"]
Oct 30 06:23:10.546768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:10.546738    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"]
Oct 30 06:23:10.663118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:10.663080    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:10.810450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:10.810346    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:10.950690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:10.950662    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:12.263972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:12.263929    5250 patch_prober.go:29] interesting pod/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:23:12.264388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:12.264006    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:23:12.275592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:12.275562    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:12.826606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:12.826568    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:12.962155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:12.962090    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:13.098479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:13.098393    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:13.452525 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:13.452492    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:14.400714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.400651    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:23:14.499238 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499169    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-podinfo\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499315    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-data\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499381    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-kube-api-access-7qsrq\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-kube-api-access-7qsrq\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499444    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-token\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-token\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499479    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-credential-socket\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499547    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-certs\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499576    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-envoy\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499613    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-data" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499630    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istiod-ca-cert\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.499700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499672    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-socket\") pod \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\" (UID: \"674faa9d-8c41-44bd-8d47-07c8c671fdc5\") "
Oct 30 06:23:14.500205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.499834    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-certs" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:14.500205 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.500069    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-socket" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:14.500349 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.500310    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-credential-socket" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:14.500431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.500393    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istiod-ca-cert" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:14.500510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.500485    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.500510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.500512    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.500707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.500528    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.500707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.500542    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.501660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.501633    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-podinfo" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:23:14.501831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.501805    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-token" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:14.501956 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.501936    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-kube-api-access-7qsrq" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "kube-api-access-7qsrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:14.502110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.502087    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-envoy" pod "674faa9d-8c41-44bd-8d47-07c8c671fdc5" (UID: "674faa9d-8c41-44bd-8d47-07c8c671fdc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:14.517117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.517091    5250 generic.go:356] "Generic (PLEG): container finished" podID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerID="efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750" exitCode=0
Oct 30 06:23:14.517216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.517143    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"
Oct 30 06:23:14.517216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.517161    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" event={"ID":"674faa9d-8c41-44bd-8d47-07c8c671fdc5","Type":"ContainerDied","Data":"efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750"}
Oct 30 06:23:14.517288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.517240    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5" event={"ID":"674faa9d-8c41-44bd-8d47-07c8c671fdc5","Type":"ContainerDied","Data":"091f86378d2fc7114815061adb484432b30fc301f8df955f396a3176cf40a0f0"}
Oct 30 06:23:14.517288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.517264    5250 scope.go:122] "RemoveContainer" containerID="efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750"
Oct 30 06:23:14.528927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.528909    5250 scope.go:122] "RemoveContainer" containerID="19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565"
Oct 30 06:23:14.539090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.539072    5250 scope.go:122] "RemoveContainer" containerID="efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750"
Oct 30 06:23:14.539395 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:14.539373    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750\": container with ID starting with efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750 not found: ID does not exist" containerID="efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750"
Oct 30 06:23:14.539479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.539406    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750"} err="failed to get container status \"efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750\": rpc error: code = NotFound desc = could not find container \"efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750\": container with ID starting with efbb880195561361081d0fd0077a3bebc4a0abc5e0df963df3da58c8bf2f7750 not found: ID does not exist"
Oct 30 06:23:14.539479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.539428    5250 scope.go:122] "RemoveContainer" containerID="19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565"
Oct 30 06:23:14.539673 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:14.539654    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565\": container with ID starting with 19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565 not found: ID does not exist" containerID="19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565"
Oct 30 06:23:14.539713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.539679    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565"} err="failed to get container status \"19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565\": rpc error: code = NotFound desc = could not find container \"19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565\": container with ID starting with 19d11154ad20f4c358080c5a4755d8d8cd941ec7d251c1bb5e136c26101a5565 not found: ID does not exist"
Oct 30 06:23:14.569696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.569669    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"]
Oct 30 06:23:14.574459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.574439    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d4bace-openshift-default-784687d7d7-2lfr5"]
Oct 30 06:23:14.601575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.601548    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qsrq\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-kube-api-access-7qsrq\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.601575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.601573    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.601726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.601582    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.601726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.601590    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.601726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.601598    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674faa9d-8c41-44bd-8d47-07c8c671fdc5-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:14.794159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:14.794019    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:15.281300 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.281265    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" path="/var/lib/kubelet/pods/674faa9d-8c41-44bd-8d47-07c8c671fdc5/volumes"
Oct 30 06:23:15.517847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.517808    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4" podUID="61e1c6f7-eadf-4103-8708-8ec58c98146f" containerName="payload-processing" containerID="cri-o://b6b253f3ae5cb373149dafbb17186783158c89ceb380359dd69783b09c5f02e8" gracePeriod=25
Oct 30 06:23:15.547632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.547555    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" podUID="d929b643-076d-4366-8b66-6093f79d44ff" containerName="payload-pre-processing" containerID="cri-o://e1b69b5fb0f701318b105cd007732a71fa5d09a0e9526e5194eefc7a30534c04" gracePeriod=25
Oct 30 06:23:15.665363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.665327    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" podUID="d929b643-076d-4366-8b66-6093f79d44ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:15.691541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.691519    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 06:23:15.713668 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.713638    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2a746e-35d3-408d-b9c1-66890f8ad655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a746e-35d3-408d-b9c1-66890f8ad655-plugins-config-volume\") pod \"ac2a746e-35d3-408d-b9c1-66890f8ad655\" (UID: \"ac2a746e-35d3-408d-b9c1-66890f8ad655\") "
Oct 30 06:23:15.713781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.713694    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2a746e-35d3-408d-b9c1-66890f8ad655-kube-api-access-zgcxz\" (UniqueName: \"kubernetes.io/projected/ac2a746e-35d3-408d-b9c1-66890f8ad655-kube-api-access-zgcxz\") pod \"ac2a746e-35d3-408d-b9c1-66890f8ad655\" (UID: \"ac2a746e-35d3-408d-b9c1-66890f8ad655\") "
Oct 30 06:23:15.713978 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.713958    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2a746e-35d3-408d-b9c1-66890f8ad655-plugins-config-volume" pod "ac2a746e-35d3-408d-b9c1-66890f8ad655" (UID: "ac2a746e-35d3-408d-b9c1-66890f8ad655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:15.716018 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.715993    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2a746e-35d3-408d-b9c1-66890f8ad655-kube-api-access-zgcxz" pod "ac2a746e-35d3-408d-b9c1-66890f8ad655" (UID: "ac2a746e-35d3-408d-b9c1-66890f8ad655"). InnerVolumeSpecName "kube-api-access-zgcxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:15.815160 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.815081    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a746e-35d3-408d-b9c1-66890f8ad655-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:15.815160 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:15.815112    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgcxz\" (UniqueName: \"kubernetes.io/projected/ac2a746e-35d3-408d-b9c1-66890f8ad655-kube-api-access-zgcxz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:16.427388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.427351    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:16.535209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.535151    5250 generic.go:356] "Generic (PLEG): container finished" podID="ac2a746e-35d3-408d-b9c1-66890f8ad655" containerID="1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc" exitCode=0
Oct 30 06:23:16.535665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.535211    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" event={"ID":"ac2a746e-35d3-408d-b9c1-66890f8ad655","Type":"ContainerDied","Data":"1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc"}
Oct 30 06:23:16.535665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.535230    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"
Oct 30 06:23:16.535665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.535271    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" event={"ID":"ac2a746e-35d3-408d-b9c1-66890f8ad655","Type":"ContainerDied","Data":"2f9ac27b935ea9fab96bc02f4b2e299f12ff33b981a48817a65c92dc2b8612b9"}
Oct 30 06:23:16.535665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.535299    5250 scope.go:122] "RemoveContainer" containerID="1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc"
Oct 30 06:23:16.546644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.546629    5250 scope.go:122] "RemoveContainer" containerID="1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc"
Oct 30 06:23:16.546923 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:16.546904    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc\": container with ID starting with 1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc not found: ID does not exist" containerID="1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc"
Oct 30 06:23:16.546982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.546931    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc"} err="failed to get container status \"1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc\": rpc error: code = NotFound desc = could not find container \"1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc\": container with ID starting with 1cc114ccb3ccd5394c0c4a0c4e030b74f27a9d8fda777702d00a05d2e566f3bc not found: ID does not exist"
Oct 30 06:23:16.587053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.587027    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"]
Oct 30 06:23:16.600878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.600849    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc"]
Oct 30 06:23:16.626857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.626814    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3abe42-7bc8c45b-zj6fc" podUID="ac2a746e-35d3-408d-b9c1-66890f8ad655" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.121:50052\" within 1s: context deadline exceeded"
Oct 30 06:23:16.627028 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:23:16.626884    5250 logging.go:55] [core] [Channel #11372 SubChannel #11373]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.121:50052", ServerName: "10.133.0.121:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.121:50052: operation was canceled"
Oct 30 06:23:16.672719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:16.672686    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4" podUID="61e1c6f7-eadf-4103-8708-8ec58c98146f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:17.281197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:17.281148    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2a746e-35d3-408d-b9c1-66890f8ad655" path="/var/lib/kubelet/pods/ac2a746e-35d3-408d-b9c1-66890f8ad655/volumes"
Oct 30 06:23:17.746997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:17.746964    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:17.924669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:17.924633    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:18.130398 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:18.130295    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:18.348598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:18.348563    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:18.438160 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:18.438128    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"]
Oct 30 06:23:18.448711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:18.448675    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-718730-lcrwj"]
Oct 30 06:23:18.493377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:18.493347    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:18.627046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:18.627014    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:18.986166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:18.986130    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:19.280968 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:19.280883    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c77a6f10-83c6-4b28-bc30-c7f9ca5cb846" path="/var/lib/kubelet/pods/c77a6f10-83c6-4b28-bc30-c7f9ca5cb846/volumes"
Oct 30 06:23:19.921356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:19.921322    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:19.976864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:19.976826    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"]
Oct 30 06:23:20.003754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.003721    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"]
Oct 30 06:23:20.022979 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.022947    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main" containerID="cri-o://5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736" gracePeriod=45
Oct 30 06:23:20.071471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.071443    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:20.223485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.223445    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:20.344716 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.344687    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 06:23:20.365049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.364980    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-model-cache\") pod \"1bc6c877-c2ee-4620-a543-e740ed43437e\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") "
Oct 30 06:23:20.365049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365030    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-tmp-dir\") pod \"1bc6c877-c2ee-4620-a543-e740ed43437e\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") "
Oct 30 06:23:20.365421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365083    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc6c877-c2ee-4620-a543-e740ed43437e-kube-api-access-qddtb\" (UniqueName: \"kubernetes.io/projected/1bc6c877-c2ee-4620-a543-e740ed43437e-kube-api-access-qddtb\") pod \"1bc6c877-c2ee-4620-a543-e740ed43437e\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") "
Oct 30 06:23:20.365421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365141    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc6c877-c2ee-4620-a543-e740ed43437e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc6c877-c2ee-4620-a543-e740ed43437e-tls-certs\") pod \"1bc6c877-c2ee-4620-a543-e740ed43437e\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") "
Oct 30 06:23:20.365421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365254    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-home\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-home\") pod \"1bc6c877-c2ee-4620-a543-e740ed43437e\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") "
Oct 30 06:23:20.365421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365293    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-dshm\") pod \"1bc6c877-c2ee-4620-a543-e740ed43437e\" (UID: \"1bc6c877-c2ee-4620-a543-e740ed43437e\") "
Oct 30 06:23:20.365421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365373    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-model-cache" pod "1bc6c877-c2ee-4620-a543-e740ed43437e" (UID: "1bc6c877-c2ee-4620-a543-e740ed43437e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:20.365716 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365419    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-tmp-dir" pod "1bc6c877-c2ee-4620-a543-e740ed43437e" (UID: "1bc6c877-c2ee-4620-a543-e740ed43437e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:20.365716 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365524    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-home" pod "1bc6c877-c2ee-4620-a543-e740ed43437e" (UID: "1bc6c877-c2ee-4620-a543-e740ed43437e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:20.365838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365782    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:20.365838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365798    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:20.365838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.365807    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:20.366312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.366275    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:20.368152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.368123    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc6c877-c2ee-4620-a543-e740ed43437e-tls-certs" pod "1bc6c877-c2ee-4620-a543-e740ed43437e" (UID: "1bc6c877-c2ee-4620-a543-e740ed43437e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:23:20.368284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.368166    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc6c877-c2ee-4620-a543-e740ed43437e-kube-api-access-qddtb" pod "1bc6c877-c2ee-4620-a543-e740ed43437e" (UID: "1bc6c877-c2ee-4620-a543-e740ed43437e"). InnerVolumeSpecName "kube-api-access-qddtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:20.368857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.368841    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-dshm" pod "1bc6c877-c2ee-4620-a543-e740ed43437e" (UID: "1bc6c877-c2ee-4620-a543-e740ed43437e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:20.466535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.466453    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qddtb\" (UniqueName: \"kubernetes.io/projected/1bc6c877-c2ee-4620-a543-e740ed43437e-kube-api-access-qddtb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:20.466535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.466480    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc6c877-c2ee-4620-a543-e740ed43437e-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:20.466535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.466489    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc6c877-c2ee-4620-a543-e740ed43437e-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:20.566950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.566914    5250 generic.go:356] "Generic (PLEG): container finished" podID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerID="5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736" exitCode=0
Oct 30 06:23:20.567141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.566977    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"
Oct 30 06:23:20.567141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.566999    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" event={"ID":"1bc6c877-c2ee-4620-a543-e740ed43437e","Type":"ContainerDied","Data":"5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736"}
Oct 30 06:23:20.567141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.567047    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7" event={"ID":"1bc6c877-c2ee-4620-a543-e740ed43437e","Type":"ContainerDied","Data":"9ffdbfb5c46a5b60eb0db07eaac374acc72d3dc9d382e7823090fd1c6b40a2a5"}
Oct 30 06:23:20.567141 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.567072    5250 scope.go:122] "RemoveContainer" containerID="5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736"
Oct 30 06:23:20.578165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.578146    5250 scope.go:122] "RemoveContainer" containerID="5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736"
Oct 30 06:23:20.578469 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:20.578451    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736\": container with ID starting with 5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736 not found: ID does not exist" containerID="5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736"
Oct 30 06:23:20.578542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.578483    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736"} err="failed to get container status \"5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736\": rpc error: code = NotFound desc = could not find container \"5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736\": container with ID starting with 5a814c59da6d4cf49ad1f656339bddb4d8db939b99687e7a9a89a51caec86736 not found: ID does not exist"
Oct 30 06:23:20.639107 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.639058    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"]
Oct 30 06:23:20.643309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:20.643269    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/simulator-w3-fffaa8-kserve-bf469b6-9rgn7"]
Oct 30 06:23:21.279888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:21.279857    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" path="/var/lib/kubelet/pods/1bc6c877-c2ee-4620-a543-e740ed43437e/volumes"
Oct 30 06:23:21.973770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:21.973715    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:23.697453 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:23.697421    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:24.977757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:24.977715    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g" podUID="3313f229-0830-4703-bc0a-0419dde5f501" containerName="payload-processing" containerID="cri-o://9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353" gracePeriod=25
Oct 30 06:23:25.004390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:25.004344    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh" podUID="0f40f894-6054-4b5d-93d3-b8a91eb6abc8" containerName="payload-pre-processing" containerID="cri-o://0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4" gracePeriod=25
Oct 30 06:23:25.043009 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:25.042979    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:25.183746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:25.183708    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:25.327689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:25.327584    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:25.476373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:25.476324    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:25.664521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:25.664478    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" podUID="d929b643-076d-4366-8b66-6093f79d44ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:26.672003 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.671963    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4" podUID="61e1c6f7-eadf-4103-8708-8ec58c98146f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:26.786972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.786928    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"]
Oct 30 06:23:26.787496 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787482    5250 cpu_manager.go:414] "containerMap: removing container" podUID="7f215d4a-d20f-4d69-9606-7e0980b3c4a2" containerName="revoke-keys"
Oct 30 06:23:26.787558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787497    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f215d4a-d20f-4d69-9606-7e0980b3c4a2" containerName="revoke-keys"
Oct 30 06:23:26.787558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787514    5250 cpu_manager.go:414] "containerMap: removing container" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy"
Oct 30 06:23:26.787558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787520    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy"
Oct 30 06:23:26.787558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787528    5250 cpu_manager.go:414] "containerMap: removing container" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy"
Oct 30 06:23:26.787558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787534    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy"
Oct 30 06:23:26.787558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787547    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ac2a746e-35d3-408d-b9c1-66890f8ad655" containerName="payload-processing"
Oct 30 06:23:26.787558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787552    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2a746e-35d3-408d-b9c1-66890f8ad655" containerName="payload-processing"
Oct 30 06:23:26.787558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787559    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787563    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787573    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787577    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787592    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787597    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787603    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787608    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787614    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787618    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787625    5250 cpu_manager.go:414] "containerMap: removing container" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787631    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787698    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787710    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6bec144-e4ab-4209-98a2-019bdfed2fe6" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787716    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b79e875-5537-492e-aa94-3a015701a0f8" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787722    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787728    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae41e05-46f8-418b-ab84-968a0e5a23b1" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787734    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f215d4a-d20f-4d69-9606-7e0980b3c4a2" containerName="revoke-keys"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787740    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787747    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2a746e-35d3-408d-b9c1-66890f8ad655" containerName="payload-processing"
Oct 30 06:23:26.787832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.787752    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc6c877-c2ee-4620-a543-e740ed43437e" containerName="main"
Oct 30 06:23:26.791132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.791117    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.810821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.810786    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"]
Oct 30 06:23:26.819109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.819086    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:23:26.819290 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.819161    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:23:26.859228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.859192    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.859449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.859304    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkrm\" (UniqueName: \"kubernetes.io/projected/36c27c2b-6cdb-4260-b468-d8cb300d7800-kube-api-access-cmkrm\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.859449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.859375    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.917495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.917456    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:26.960552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.960446    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.960552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.960486    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkrm\" (UniqueName: \"kubernetes.io/projected/36c27c2b-6cdb-4260-b468-d8cb300d7800-kube-api-access-cmkrm\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.960806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.960615    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.961284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.961259    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.963396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.963370    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:26.984394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:26.984361    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkrm\" (UniqueName: \"kubernetes.io/projected/36c27c2b-6cdb-4260-b468-d8cb300d7800-kube-api-access-cmkrm\") pod \"maas-api-revoke-sub-a137847f-d5g29\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:27.105576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:27.105543    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:27.425472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:27.425432    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"]
Oct 30 06:23:27.429230 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:23:27.429154    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c27c2b_6cdb_4260_b468_d8cb300d7800.slice/crio-d3c414fde089f3a51d3ee9b01023523e7a37d20c4751bded3e8f856b628d4cfa WatchSource:0}: Error finding container d3c414fde089f3a51d3ee9b01023523e7a37d20c4751bded3e8f856b628d4cfa: Status 404 returned error can't find the container with id d3c414fde089f3a51d3ee9b01023523e7a37d20c4751bded3e8f856b628d4cfa
Oct 30 06:23:27.617721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:27.617681    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29" event={"ID":"36c27c2b-6cdb-4260-b468-d8cb300d7800","Type":"ContainerStarted","Data":"d3c414fde089f3a51d3ee9b01023523e7a37d20c4751bded3e8f856b628d4cfa"}
Oct 30 06:23:27.647162 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:27.647134    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:27.849914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:27.849880    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:28.263051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:28.262984    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:28.626566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:28.626478    5250 generic.go:356] "Generic (PLEG): container finished" podID="36c27c2b-6cdb-4260-b468-d8cb300d7800" containerID="a242f9633cc71319a4fe0663e36140e24453a970014dad46968b0c79075a1101" exitCode=0
Oct 30 06:23:28.626566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:28.626510    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29" event={"ID":"36c27c2b-6cdb-4260-b468-d8cb300d7800","Type":"ContainerDied","Data":"a242f9633cc71319a4fe0663e36140e24453a970014dad46968b0c79075a1101"}
Oct 30 06:23:29.385660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.385626    5250 scope.go:122] "RemoveContainer" containerID="385037203e921bbc752e3b63078b63cec50bd0d01a2d9eff0191883dcd9c87bf"
Oct 30 06:23:29.396137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.396115    5250 scope.go:122] "RemoveContainer" containerID="c65c9f7fa73b174b3a0c2b011dc9098d30880ade92893084895fe6720ff71e9b"
Oct 30 06:23:29.573700 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.573659    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:29.784939 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.784910    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:29.894589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.894501    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c27c2b-6cdb-4260-b468-d8cb300d7800-kube-api-access-cmkrm\" (UniqueName: \"kubernetes.io/projected/36c27c2b-6cdb-4260-b468-d8cb300d7800-kube-api-access-cmkrm\") pod \"36c27c2b-6cdb-4260-b468-d8cb300d7800\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") "
Oct 30 06:23:29.894754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.894636    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-xks-ca\") pod \"36c27c2b-6cdb-4260-b468-d8cb300d7800\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") "
Oct 30 06:23:29.894754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.894702    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-openshift-ca\") pod \"36c27c2b-6cdb-4260-b468-d8cb300d7800\" (UID: \"36c27c2b-6cdb-4260-b468-d8cb300d7800\") "
Oct 30 06:23:29.895117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.895088    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-openshift-ca" pod "36c27c2b-6cdb-4260-b468-d8cb300d7800" (UID: "36c27c2b-6cdb-4260-b468-d8cb300d7800"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:29.896764 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.896743    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-xks-ca" pod "36c27c2b-6cdb-4260-b468-d8cb300d7800" (UID: "36c27c2b-6cdb-4260-b468-d8cb300d7800"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:23:29.896885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.896865    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c27c2b-6cdb-4260-b468-d8cb300d7800-kube-api-access-cmkrm" pod "36c27c2b-6cdb-4260-b468-d8cb300d7800" (UID: "36c27c2b-6cdb-4260-b468-d8cb300d7800"). InnerVolumeSpecName "kube-api-access-cmkrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:29.996278 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.996242    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:29.996278 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.996271    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c27c2b-6cdb-4260-b468-d8cb300d7800-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:29.996461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:29.996285    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkrm\" (UniqueName: \"kubernetes.io/projected/36c27c2b-6cdb-4260-b468-d8cb300d7800-kube-api-access-cmkrm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:30.640493 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:30.640462    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29" event={"ID":"36c27c2b-6cdb-4260-b468-d8cb300d7800","Type":"ContainerDied","Data":"d3c414fde089f3a51d3ee9b01023523e7a37d20c4751bded3e8f856b628d4cfa"}
Oct 30 06:23:30.640894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:30.640510    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3c414fde089f3a51d3ee9b01023523e7a37d20c4751bded3e8f856b628d4cfa"
Oct 30 06:23:30.640894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:30.640471    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"
Oct 30 06:23:31.067263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:31.067169    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"]
Oct 30 06:23:31.090878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:31.090847    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"]
Oct 30 06:23:31.215150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:31.215120    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:31.363674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:31.363592    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:32.374059 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:32.374025    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:23:32.720359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:32.720319    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:33.576167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:33.576130    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g" podUID="3313f229-0830-4703-bc0a-0419dde5f501" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:34.322754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:34.322711    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:34.464880 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:34.464833    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:34.588201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:34.588094    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh" podUID="0f40f894-6054-4b5d-93d3-b8a91eb6abc8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:34.600094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:34.600066    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"]
Oct 30 06:23:34.600359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:34.600321    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy" containerID="cri-o://8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a" gracePeriod=30
Oct 30 06:23:34.672657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:34.672606    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:35.663709 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:23:35.663683    5250 logging.go:55] [core] [Channel #11452 SubChannel #11453]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.44:50052", ServerName: "10.133.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.44:50052: connect: connection refused"
Oct 30 06:23:35.676677 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.676625    5250 generic.go:356] "Generic (PLEG): container finished" podID="61e1c6f7-eadf-4103-8708-8ec58c98146f" containerID="b6b253f3ae5cb373149dafbb17186783158c89ceb380359dd69783b09c5f02e8" exitCode=0
Oct 30 06:23:35.676817 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.676713    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4" event={"ID":"61e1c6f7-eadf-4103-8708-8ec58c98146f","Type":"ContainerDied","Data":"b6b253f3ae5cb373149dafbb17186783158c89ceb380359dd69783b09c5f02e8"}
Oct 30 06:23:35.678761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.678741    5250 generic.go:356] "Generic (PLEG): container finished" podID="d929b643-076d-4366-8b66-6093f79d44ff" containerID="e1b69b5fb0f701318b105cd007732a71fa5d09a0e9526e5194eefc7a30534c04" exitCode=0
Oct 30 06:23:35.678899 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.678812    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" event={"ID":"d929b643-076d-4366-8b66-6093f79d44ff","Type":"ContainerDied","Data":"e1b69b5fb0f701318b105cd007732a71fa5d09a0e9526e5194eefc7a30534c04"}
Oct 30 06:23:35.819371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.819346    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 06:23:35.848335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.848312    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 06:23:35.964078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.964039    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d929b643-076d-4366-8b66-6093f79d44ff-kube-api-access-j6l9r\" (UniqueName: \"kubernetes.io/projected/d929b643-076d-4366-8b66-6093f79d44ff-kube-api-access-j6l9r\") pod \"d929b643-076d-4366-8b66-6093f79d44ff\" (UID: \"d929b643-076d-4366-8b66-6093f79d44ff\") "
Oct 30 06:23:35.964078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.964082    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e1c6f7-eadf-4103-8708-8ec58c98146f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1c6f7-eadf-4103-8708-8ec58c98146f-plugins-config-volume\") pod \"61e1c6f7-eadf-4103-8708-8ec58c98146f\" (UID: \"61e1c6f7-eadf-4103-8708-8ec58c98146f\") "
Oct 30 06:23:35.964390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.964205    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d929b643-076d-4366-8b66-6093f79d44ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d929b643-076d-4366-8b66-6093f79d44ff-plugins-config-volume\") pod \"d929b643-076d-4366-8b66-6093f79d44ff\" (UID: \"d929b643-076d-4366-8b66-6093f79d44ff\") "
Oct 30 06:23:35.964390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.964251    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e1c6f7-eadf-4103-8708-8ec58c98146f-kube-api-access-p2gqw\" (UniqueName: \"kubernetes.io/projected/61e1c6f7-eadf-4103-8708-8ec58c98146f-kube-api-access-p2gqw\") pod \"61e1c6f7-eadf-4103-8708-8ec58c98146f\" (UID: \"61e1c6f7-eadf-4103-8708-8ec58c98146f\") "
Oct 30 06:23:35.964570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.964544    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e1c6f7-eadf-4103-8708-8ec58c98146f-plugins-config-volume" pod "61e1c6f7-eadf-4103-8708-8ec58c98146f" (UID: "61e1c6f7-eadf-4103-8708-8ec58c98146f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:35.964625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.964577    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d929b643-076d-4366-8b66-6093f79d44ff-plugins-config-volume" pod "d929b643-076d-4366-8b66-6093f79d44ff" (UID: "d929b643-076d-4366-8b66-6093f79d44ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:35.964826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.964809    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e1c6f7-eadf-4103-8708-8ec58c98146f-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:35.964892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.964829    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d929b643-076d-4366-8b66-6093f79d44ff-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:35.966501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.966483    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d929b643-076d-4366-8b66-6093f79d44ff-kube-api-access-j6l9r" pod "d929b643-076d-4366-8b66-6093f79d44ff" (UID: "d929b643-076d-4366-8b66-6093f79d44ff"). InnerVolumeSpecName "kube-api-access-j6l9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:35.966565 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:35.966538    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e1c6f7-eadf-4103-8708-8ec58c98146f-kube-api-access-p2gqw" pod "61e1c6f7-eadf-4103-8708-8ec58c98146f" (UID: "61e1c6f7-eadf-4103-8708-8ec58c98146f"). InnerVolumeSpecName "kube-api-access-p2gqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:36.065703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.065626    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6l9r\" (UniqueName: \"kubernetes.io/projected/d929b643-076d-4366-8b66-6093f79d44ff-kube-api-access-j6l9r\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:36.065703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.065653    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gqw\" (UniqueName: \"kubernetes.io/projected/61e1c6f7-eadf-4103-8708-8ec58c98146f-kube-api-access-p2gqw\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:36.068025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.068004    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx" podUID="ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" containerName="payload-processing" containerID="cri-o://12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5" gracePeriod=25
Oct 30 06:23:36.091707 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.091665    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn" podUID="fcf5f568-1f52-426e-94a4-aece3c315cda" containerName="payload-pre-processing" containerID="cri-o://5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f" gracePeriod=25
Oct 30 06:23:36.197008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.196976    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"]
Oct 30 06:23:36.221805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.221772    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"]
Oct 30 06:23:36.223620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.223579    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:36.368169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.368065    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:36.515439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.515403    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:36.664353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.664302    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" podUID="d929b643-076d-4366-8b66-6093f79d44ff" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.44:50052\" within 1s: context deadline exceeded"
Oct 30 06:23:36.664786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.664392    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 06:23:36.664786 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:23:36.664434    5250 logging.go:55] [core] [Channel #11452 SubChannel #11453]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.44:50052", ServerName: "10.133.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.44:50052: operation was canceled"
Oct 30 06:23:36.689725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.688809    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"
Oct 30 06:23:36.689725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.688841    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4" event={"ID":"61e1c6f7-eadf-4103-8708-8ec58c98146f","Type":"ContainerDied","Data":"ffdf53a87877dc799d4441aa9aa6785a8e38ee75200b349fe0a888c3807e4b53"}
Oct 30 06:23:36.689725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.688986    5250 scope.go:122] "RemoveContainer" containerID="b6b253f3ae5cb373149dafbb17186783158c89ceb380359dd69783b09c5f02e8"
Oct 30 06:23:36.693894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.693865    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk" event={"ID":"d929b643-076d-4366-8b66-6093f79d44ff","Type":"ContainerDied","Data":"c3ca94ad6d4a27cc19111954dd1146fbcf21eaca55d935529af1f05098b5cc3c"}
Oct 30 06:23:36.694028 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.693981    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"
Oct 30 06:23:36.722962 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.722271    5250 scope.go:122] "RemoveContainer" containerID="e1b69b5fb0f701318b105cd007732a71fa5d09a0e9526e5194eefc7a30534c04"
Oct 30 06:23:36.763433 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.763401    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"]
Oct 30 06:23:36.768966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.768937    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fffaa8-7cdd9459c9-wrdzk"]
Oct 30 06:23:36.803078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.803046    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"]
Oct 30 06:23:36.808737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:36.808713    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fffaa8-f7fcbc756-g2xx4"]
Oct 30 06:23:37.206330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:37.206297    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx" podUID="ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:37.281351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:37.281321    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e1c6f7-eadf-4103-8708-8ec58c98146f" path="/var/lib/kubelet/pods/61e1c6f7-eadf-4103-8708-8ec58c98146f/volumes"
Oct 30 06:23:37.281743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:37.281729    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d929b643-076d-4366-8b66-6093f79d44ff" path="/var/lib/kubelet/pods/d929b643-076d-4366-8b66-6093f79d44ff/volumes"
Oct 30 06:23:37.477057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:37.476969    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:37.861047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:37.860952    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:37.994872 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:37.994829    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn" podUID="fcf5f568-1f52-426e-94a4-aece3c315cda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:39.487730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.487598    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:39.865560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.865531    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:23:39.911094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911063    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c353321f-4449-4038-9811-5edea86b7ed5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c353321f-4449-4038-9811-5edea86b7ed5-istio-podinfo\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:39.911281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911112    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c353321f-4449-4038-9811-5edea86b7ed5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c353321f-4449-4038-9811-5edea86b7ed5-istiod-ca-cert\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:39.911281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911163    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-certs\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:39.911281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911215    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-socket\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:39.911281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911275    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-envoy\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:39.911510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911316    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-credential-socket\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:39.911510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911347    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-kube-api-access-9pbm7\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-kube-api-access-9pbm7\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:39.911510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911399    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-istio-token\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-istio-token\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:39.911673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911548    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-certs" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:39.911673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911563    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-socket" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:39.911673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911573    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c353321f-4449-4038-9811-5edea86b7ed5-istiod-ca-cert" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:39.911673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911659    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-credential-socket" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:39.911878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911861    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:39.911938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911881    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c353321f-4449-4038-9811-5edea86b7ed5-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:39.911938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911897    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:39.911938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.911911    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:39.913563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.913537    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-envoy" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:39.913563 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.913546    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c353321f-4449-4038-9811-5edea86b7ed5-istio-podinfo" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:23:39.913738 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.913698    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-kube-api-access-9pbm7" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "kube-api-access-9pbm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:39.913842 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:39.913813    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-istio-token" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:40.012462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.012432    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-data\") pod \"c353321f-4449-4038-9811-5edea86b7ed5\" (UID: \"c353321f-4449-4038-9811-5edea86b7ed5\") "
Oct 30 06:23:40.012730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.012708    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-data" pod "c353321f-4449-4038-9811-5edea86b7ed5" (UID: "c353321f-4449-4038-9811-5edea86b7ed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:40.012803 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.012788    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:40.012843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.012810    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbm7\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-kube-api-access-9pbm7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:40.012843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.012821    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c353321f-4449-4038-9811-5edea86b7ed5-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:40.012843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.012829    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c353321f-4449-4038-9811-5edea86b7ed5-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:40.012843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.012836    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c353321f-4449-4038-9811-5edea86b7ed5-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:40.725507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.725471    5250 generic.go:356] "Generic (PLEG): container finished" podID="c353321f-4449-4038-9811-5edea86b7ed5" containerID="8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a" exitCode=0
Oct 30 06:23:40.725960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.725539    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"
Oct 30 06:23:40.725960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.725539    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" event={"ID":"c353321f-4449-4038-9811-5edea86b7ed5","Type":"ContainerDied","Data":"8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a"}
Oct 30 06:23:40.725960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.725661    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn" event={"ID":"c353321f-4449-4038-9811-5edea86b7ed5","Type":"ContainerDied","Data":"3ccf70e66777dc22aa55f7c0a1bb0b781b6af79df75a76393c4f23a91f40e281"}
Oct 30 06:23:40.725960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.725685    5250 scope.go:122] "RemoveContainer" containerID="8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a"
Oct 30 06:23:40.737587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.737565    5250 scope.go:122] "RemoveContainer" containerID="2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73"
Oct 30 06:23:40.748588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.748571    5250 scope.go:122] "RemoveContainer" containerID="8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a"
Oct 30 06:23:40.748684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.748611    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:40.748923 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:40.748905    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a\": container with ID starting with 8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a not found: ID does not exist" containerID="8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a"
Oct 30 06:23:40.748995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.748939    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a"} err="failed to get container status \"8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a\": rpc error: code = NotFound desc = could not find container \"8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a\": container with ID starting with 8a2502879f6f56be748d6e7af37749c6ba657715d592980f0e1d4dd5b3610c2a not found: ID does not exist"
Oct 30 06:23:40.748995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.748955    5250 scope.go:122] "RemoveContainer" containerID="2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73"
Oct 30 06:23:40.749465 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:40.749444    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73\": container with ID starting with 2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73 not found: ID does not exist" containerID="2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73"
Oct 30 06:23:40.749575 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.749470    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73"} err="failed to get container status \"2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73\": rpc error: code = NotFound desc = could not find container \"2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73\": container with ID starting with 2397dcd249b8f13dd71d3581872ea95f91c75885573d3a8efe8969a282cbfb73 not found: ID does not exist"
Oct 30 06:23:40.777855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.777825    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"]
Oct 30 06:23:40.789871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:40.789844    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-720e9c-openshift-default-668cd59c67-nl8wn"]
Oct 30 06:23:41.011586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.011492    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"]
Oct 30 06:23:41.011773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.011738    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582" podUID="c1c2e54f-5ae3-4276-9a26-67874c35e80a" containerName="maas-api" containerID="cri-o://db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a" gracePeriod=30
Oct 30 06:23:41.172016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.171978    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:41.198013 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.197958    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5" podUID="c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" containerName="payload-processing" containerID="cri-o://2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b" gracePeriod=25
Oct 30 06:23:41.223201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.222157    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj" podUID="b56bf06b-54bc-4ba4-811b-111b4b02bd0c" containerName="payload-pre-processing" containerID="cri-o://66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5" gracePeriod=25
Oct 30 06:23:41.280960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.280929    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 06:23:41.281923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.281894    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c353321f-4449-4038-9811-5edea86b7ed5" path="/var/lib/kubelet/pods/c353321f-4449-4038-9811-5edea86b7ed5/volumes"
Oct 30 06:23:41.308728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.308703    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"]
Oct 30 06:23:41.311344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.311320    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fffaa8-zmh7r"]
Oct 30 06:23:41.326027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.326000    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:41.327084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.327058    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls\") pod \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") "
Oct 30 06:23:41.327239 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.327105    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c2e54f-5ae3-4276-9a26-67874c35e80a-kube-api-access-brcxd\" (UniqueName: \"kubernetes.io/projected/c1c2e54f-5ae3-4276-9a26-67874c35e80a-kube-api-access-brcxd\") pod \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") "
Oct 30 06:23:41.327239 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.327171    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-metrics-certs\") pod \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") "
Oct 30 06:23:41.327373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.327249    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c2e54f-5ae3-4276-9a26-67874c35e80a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2e54f-5ae3-4276-9a26-67874c35e80a-openshift-service-ca\") pod \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\" (UID: \"c1c2e54f-5ae3-4276-9a26-67874c35e80a\") "
Oct 30 06:23:41.327836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.327805    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c2e54f-5ae3-4276-9a26-67874c35e80a-openshift-service-ca" pod "c1c2e54f-5ae3-4276-9a26-67874c35e80a" (UID: "c1c2e54f-5ae3-4276-9a26-67874c35e80a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:41.330236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.330204    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c2e54f-5ae3-4276-9a26-67874c35e80a-kube-api-access-brcxd" pod "c1c2e54f-5ae3-4276-9a26-67874c35e80a" (UID: "c1c2e54f-5ae3-4276-9a26-67874c35e80a"). InnerVolumeSpecName "kube-api-access-brcxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:41.330236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.330223    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls" pod "c1c2e54f-5ae3-4276-9a26-67874c35e80a" (UID: "c1c2e54f-5ae3-4276-9a26-67874c35e80a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:23:41.330400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.330359    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-metrics-certs" pod "c1c2e54f-5ae3-4276-9a26-67874c35e80a" (UID: "c1c2e54f-5ae3-4276-9a26-67874c35e80a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:23:41.429129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.429093    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-maas-api-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:41.429129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.429123    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brcxd\" (UniqueName: \"kubernetes.io/projected/c1c2e54f-5ae3-4276-9a26-67874c35e80a-kube-api-access-brcxd\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:41.429383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.429137    5250 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c2e54f-5ae3-4276-9a26-67874c35e80a-metrics-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:41.429383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.429150    5250 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2e54f-5ae3-4276-9a26-67874c35e80a-openshift-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:41.667306 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.667270    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:41.736086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.736052    5250 generic.go:356] "Generic (PLEG): container finished" podID="c1c2e54f-5ae3-4276-9a26-67874c35e80a" containerID="db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a" exitCode=0
Oct 30 06:23:41.736527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.736100    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"
Oct 30 06:23:41.736527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.736131    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582" event={"ID":"c1c2e54f-5ae3-4276-9a26-67874c35e80a","Type":"ContainerDied","Data":"db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a"}
Oct 30 06:23:41.736527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.736210    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582" event={"ID":"c1c2e54f-5ae3-4276-9a26-67874c35e80a","Type":"ContainerDied","Data":"eba316533682c6bc11eb5880760c9c0318a4c612180d39c926a139c4ec79ca96"}
Oct 30 06:23:41.736527 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.736233    5250 scope.go:122] "RemoveContainer" containerID="db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a"
Oct 30 06:23:41.750850 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.750827    5250 scope.go:122] "RemoveContainer" containerID="db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a"
Oct 30 06:23:41.751208 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:41.751166    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a\": container with ID starting with db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a not found: ID does not exist" containerID="db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a"
Oct 30 06:23:41.751321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.751213    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a"} err="failed to get container status \"db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a\": rpc error: code = NotFound desc = could not find container \"db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a\": container with ID starting with db9f623c2e4d3f4a262cbdc7e177a39d02994193ccd6cdf315e5c589b915396a not found: ID does not exist"
Oct 30 06:23:41.785429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.785397    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"]
Oct 30 06:23:41.796677 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.796645    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-452a59-78444c955c-6x582"]
Oct 30 06:23:41.818811 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.815407    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:41.965869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:41.965780    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:42.301561 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:42.301475    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:43.050520 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:43.050473    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:43.202203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:43.193068    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:43.280841 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:43.280803    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f215d4a-d20f-4d69-9606-7e0980b3c4a2" path="/var/lib/kubelet/pods/7f215d4a-d20f-4d69-9606-7e0980b3c4a2/volumes"
Oct 30 06:23:43.281294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:43.281279    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c2e54f-5ae3-4276-9a26-67874c35e80a" path="/var/lib/kubelet/pods/c1c2e54f-5ae3-4276-9a26-67874c35e80a/volumes"
Oct 30 06:23:43.330081 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:43.330005    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:43.474042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:43.473994    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:43.575981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:43.575939    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g" podUID="3313f229-0830-4703-bc0a-0419dde5f501" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:44.025496 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:44.025457    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:44.172971 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:44.172938    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:44.586235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:44.586163    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh" podUID="0f40f894-6054-4b5d-93d3-b8a91eb6abc8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:44.694740 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:44.694694    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:45.260191 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.260159    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:23:45.293603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.293580    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:23:45.377305 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.377274    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-kube-api-access-7zlnj\" (UniqueName: \"kubernetes.io/projected/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-kube-api-access-7zlnj\") pod \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\" (UID: \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\") "
Oct 30 06:23:45.377504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.377360    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3313f229-0830-4703-bc0a-0419dde5f501-kube-api-access-r7qsq\" (UniqueName: \"kubernetes.io/projected/3313f229-0830-4703-bc0a-0419dde5f501-kube-api-access-r7qsq\") pod \"3313f229-0830-4703-bc0a-0419dde5f501\" (UID: \"3313f229-0830-4703-bc0a-0419dde5f501\") "
Oct 30 06:23:45.377504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.377413    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-plugins-config-volume\") pod \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\" (UID: \"0f40f894-6054-4b5d-93d3-b8a91eb6abc8\") "
Oct 30 06:23:45.377645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.377509    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3313f229-0830-4703-bc0a-0419dde5f501-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313f229-0830-4703-bc0a-0419dde5f501-plugins-config-volume\") pod \"3313f229-0830-4703-bc0a-0419dde5f501\" (UID: \"3313f229-0830-4703-bc0a-0419dde5f501\") "
Oct 30 06:23:45.377883 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.377854    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-plugins-config-volume" pod "0f40f894-6054-4b5d-93d3-b8a91eb6abc8" (UID: "0f40f894-6054-4b5d-93d3-b8a91eb6abc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:45.377999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.377854    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3313f229-0830-4703-bc0a-0419dde5f501-plugins-config-volume" pod "3313f229-0830-4703-bc0a-0419dde5f501" (UID: "3313f229-0830-4703-bc0a-0419dde5f501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:45.379709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.379689    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3313f229-0830-4703-bc0a-0419dde5f501-kube-api-access-r7qsq" pod "3313f229-0830-4703-bc0a-0419dde5f501" (UID: "3313f229-0830-4703-bc0a-0419dde5f501"). InnerVolumeSpecName "kube-api-access-r7qsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:45.379785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.379721    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-kube-api-access-7zlnj" pod "0f40f894-6054-4b5d-93d3-b8a91eb6abc8" (UID: "0f40f894-6054-4b5d-93d3-b8a91eb6abc8"). InnerVolumeSpecName "kube-api-access-7zlnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:45.458999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.458971    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"]
Oct 30 06:23:45.478679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.478648    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7qsq\" (UniqueName: \"kubernetes.io/projected/3313f229-0830-4703-bc0a-0419dde5f501-kube-api-access-r7qsq\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:45.478679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.478670    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:45.478679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.478679    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313f229-0830-4703-bc0a-0419dde5f501-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:45.478881 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.478688    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zlnj\" (UniqueName: \"kubernetes.io/projected/0f40f894-6054-4b5d-93d3-b8a91eb6abc8-kube-api-access-7zlnj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:45.488316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.488237    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"]
Oct 30 06:23:45.622283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.622250    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:45.767995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.767900    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:45.768772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.768744    5250 generic.go:356] "Generic (PLEG): container finished" podID="3313f229-0830-4703-bc0a-0419dde5f501" containerID="9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353" exitCode=0
Oct 30 06:23:45.768923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.768795    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"
Oct 30 06:23:45.768923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.768834    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g" event={"ID":"3313f229-0830-4703-bc0a-0419dde5f501","Type":"ContainerDied","Data":"9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353"}
Oct 30 06:23:45.768923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.768883    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g" event={"ID":"3313f229-0830-4703-bc0a-0419dde5f501","Type":"ContainerDied","Data":"7fcd29b9fdcde50459da2653b25e9fd66ccf524dc3cd1cd32ea663349c514802"}
Oct 30 06:23:45.768923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.768907    5250 scope.go:122] "RemoveContainer" containerID="9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353"
Oct 30 06:23:45.770164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.770136    5250 generic.go:356] "Generic (PLEG): container finished" podID="0f40f894-6054-4b5d-93d3-b8a91eb6abc8" containerID="0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4" exitCode=0
Oct 30 06:23:45.770304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.770198    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"
Oct 30 06:23:45.770304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.770228    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh" event={"ID":"0f40f894-6054-4b5d-93d3-b8a91eb6abc8","Type":"ContainerDied","Data":"0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4"}
Oct 30 06:23:45.770304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.770281    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh" event={"ID":"0f40f894-6054-4b5d-93d3-b8a91eb6abc8","Type":"ContainerDied","Data":"00307e4a8c61ef5ca03772ff9936331fb95f44ef51ec8b16f1669eec1e2cc653"}
Oct 30 06:23:45.781532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.781497    5250 scope.go:122] "RemoveContainer" containerID="9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353"
Oct 30 06:23:45.781821 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:45.781797    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353\": container with ID starting with 9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353 not found: ID does not exist" containerID="9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353"
Oct 30 06:23:45.781923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.781828    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353"} err="failed to get container status \"9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353\": rpc error: code = NotFound desc = could not find container \"9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353\": container with ID starting with 9409d4971595248b3f317976628e1800ffe3c032ac4a84d7e772b8881d670353 not found: ID does not exist"
Oct 30 06:23:45.781923 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.781844    5250 scope.go:122] "RemoveContainer" containerID="0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4"
Oct 30 06:23:45.790931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.790913    5250 scope.go:122] "RemoveContainer" containerID="0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4"
Oct 30 06:23:45.791200 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:45.791165    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4\": container with ID starting with 0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4 not found: ID does not exist" containerID="0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4"
Oct 30 06:23:45.791254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.791210    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4"} err="failed to get container status \"0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4\": rpc error: code = NotFound desc = could not find container \"0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4\": container with ID starting with 0b754898f013e8ce1846d5f2a42541087961d4e6a906fcf42aaefb371043dfe4 not found: ID does not exist"
Oct 30 06:23:45.814447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.814423    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"]
Oct 30 06:23:45.820701 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.820679    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-720e9c-57cc79f5d8-mn22g"]
Oct 30 06:23:45.849218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.849195    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"]
Oct 30 06:23:45.855629 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:45.855611    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-720e9c-7d79d758ff-bm7kh"]
Oct 30 06:23:46.101890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:46.101813    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:47.205705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:47.205665    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx" podUID="ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:47.280163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:47.280121    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f40f894-6054-4b5d-93d3-b8a91eb6abc8" path="/var/lib/kubelet/pods/0f40f894-6054-4b5d-93d3-b8a91eb6abc8/volumes"
Oct 30 06:23:47.280631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:47.280609    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3313f229-0830-4703-bc0a-0419dde5f501" path="/var/lib/kubelet/pods/3313f229-0830-4703-bc0a-0419dde5f501/volumes"
Oct 30 06:23:47.772865 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:47.769967    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:47.994576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:47.994543    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn" podUID="fcf5f568-1f52-426e-94a4-aece3c315cda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:48.999400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:48.999364    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:49.153522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:49.153483    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:49.323529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:49.323449    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:49.528132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:49.528098    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:49.727938 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:49.727903    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:49.906587 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:49.906526    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"]
Oct 30 06:23:49.906826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:49.906776    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy" containerID="cri-o://2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980" gracePeriod=30
Oct 30 06:23:49.977318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:49.977281    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj" podUID="b56bf06b-54bc-4ba4-811b-111b4b02bd0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:49.986673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:49.986603    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:50.447389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.447357    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:50.460200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.460148    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst" podUID="d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a" containerName="payload-processing" containerID="cri-o://4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889" gracePeriod=25
Oct 30 06:23:50.488643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.488609    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b" podUID="16498d8f-ea36-4a05-b012-13c320cfe29a" containerName="payload-pre-processing" containerID="cri-o://428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0" gracePeriod=25
Oct 30 06:23:50.567004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.566976    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"]
Oct 30 06:23:50.567490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567475    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3313f229-0830-4703-bc0a-0419dde5f501" containerName="payload-processing"
Oct 30 06:23:50.567543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567492    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3313f229-0830-4703-bc0a-0419dde5f501" containerName="payload-processing"
Oct 30 06:23:50.567543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567515    5250 cpu_manager.go:414] "containerMap: removing container" podUID="36c27c2b-6cdb-4260-b468-d8cb300d7800" containerName="revoke-keys"
Oct 30 06:23:50.567543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567520    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c27c2b-6cdb-4260-b468-d8cb300d7800" containerName="revoke-keys"
Oct 30 06:23:50.567543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567532    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c1c2e54f-5ae3-4276-9a26-67874c35e80a" containerName="maas-api"
Oct 30 06:23:50.567543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567537    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c2e54f-5ae3-4276-9a26-67874c35e80a" containerName="maas-api"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567549    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567555    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567564    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567568    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567578    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0f40f894-6054-4b5d-93d3-b8a91eb6abc8" containerName="payload-pre-processing"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567587    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f40f894-6054-4b5d-93d3-b8a91eb6abc8" containerName="payload-pre-processing"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567603    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d929b643-076d-4366-8b66-6093f79d44ff" containerName="payload-pre-processing"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567608    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d929b643-076d-4366-8b66-6093f79d44ff" containerName="payload-pre-processing"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567619    5250 cpu_manager.go:414] "containerMap: removing container" podUID="61e1c6f7-eadf-4103-8708-8ec58c98146f" containerName="payload-processing"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567625    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e1c6f7-eadf-4103-8708-8ec58c98146f" containerName="payload-processing"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567693    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c27c2b-6cdb-4260-b468-d8cb300d7800" containerName="revoke-keys"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567703    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c2e54f-5ae3-4276-9a26-67874c35e80a" containerName="maas-api"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567709    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567715    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c353321f-4449-4038-9811-5edea86b7ed5" containerName="istio-proxy"
Oct 30 06:23:50.567719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567724    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f40f894-6054-4b5d-93d3-b8a91eb6abc8" containerName="payload-pre-processing"
Oct 30 06:23:50.568231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567735    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d929b643-076d-4366-8b66-6093f79d44ff" containerName="payload-pre-processing"
Oct 30 06:23:50.568231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567746    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e1c6f7-eadf-4103-8708-8ec58c98146f" containerName="payload-processing"
Oct 30 06:23:50.568231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567754    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="674faa9d-8c41-44bd-8d47-07c8c671fdc5" containerName="istio-proxy"
Oct 30 06:23:50.568231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.567764    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3313f229-0830-4703-bc0a-0419dde5f501" containerName="payload-processing"
Oct 30 06:23:50.571303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.571280    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.585831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.585810    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:23:50.585944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.585892    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:23:50.590232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.590210    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"]
Oct 30 06:23:50.643726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.643691    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wx54\" (UniqueName: \"kubernetes.io/projected/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-kube-api-access-8wx54\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.643916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.643736    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.643996 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.643905    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.745549 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.745518    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.745705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.745671    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wx54\" (UniqueName: \"kubernetes.io/projected/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-kube-api-access-8wx54\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.745705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.745703    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.746363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.746339    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.748576 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.748551    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.765552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.765527    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:23:50.769097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.769071    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wx54\" (UniqueName: \"kubernetes.io/projected/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-kube-api-access-8wx54\") pod \"maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.779017 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.778995    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:23:50.810168 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.810139    5250 generic.go:356] "Generic (PLEG): container finished" podID="d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a" containerID="4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889" exitCode=0
Oct 30 06:23:50.810319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.810244    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"
Oct 30 06:23:50.810319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.810294    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst" event={"ID":"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a","Type":"ContainerDied","Data":"4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889"}
Oct 30 06:23:50.810451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.810349    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst" event={"ID":"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a","Type":"ContainerDied","Data":"35c3edb0b8488ffe0be46ddb8f0046cc342e661d35f16e13330198e5b6670be1"}
Oct 30 06:23:50.810451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.810382    5250 scope.go:122] "RemoveContainer" containerID="4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889"
Oct 30 06:23:50.811864 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.811842    5250 generic.go:356] "Generic (PLEG): container finished" podID="16498d8f-ea36-4a05-b012-13c320cfe29a" containerID="428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0" exitCode=0
Oct 30 06:23:50.811984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.811881    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"
Oct 30 06:23:50.811984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.811924    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b" event={"ID":"16498d8f-ea36-4a05-b012-13c320cfe29a","Type":"ContainerDied","Data":"428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0"}
Oct 30 06:23:50.812093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.812006    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b" event={"ID":"16498d8f-ea36-4a05-b012-13c320cfe29a","Type":"ContainerDied","Data":"7fae355992bf40945a0ce0874026a7da5a18c4a1bc2a55d6644057b22d5b68fd"}
Oct 30 06:23:50.820720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.820701    5250 scope.go:122] "RemoveContainer" containerID="4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889"
Oct 30 06:23:50.821009 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:50.820981    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889\": container with ID starting with 4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889 not found: ID does not exist" containerID="4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889"
Oct 30 06:23:50.821107 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.821019    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889"} err="failed to get container status \"4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889\": rpc error: code = NotFound desc = could not find container \"4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889\": container with ID starting with 4e4cec48096271a635de07e0c82a78cbcc259f307f4e92757c5dd1e16dd2a889 not found: ID does not exist"
Oct 30 06:23:50.821107 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.821040    5250 scope.go:122] "RemoveContainer" containerID="428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0"
Oct 30 06:23:50.832209 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.832170    5250 scope.go:122] "RemoveContainer" containerID="428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0"
Oct 30 06:23:50.832520 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:50.832497    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0\": container with ID starting with 428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0 not found: ID does not exist" containerID="428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0"
Oct 30 06:23:50.832605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.832533    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0"} err="failed to get container status \"428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0\": rpc error: code = NotFound desc = could not find container \"428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0\": container with ID starting with 428ec8ef05e16e9973420b68ec17160a64552aa1286651bfd9baee2399f5e8e0 not found: ID does not exist"
Oct 30 06:23:50.846464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.846436    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16498d8f-ea36-4a05-b012-13c320cfe29a-kube-api-access-98c9k\" (UniqueName: \"kubernetes.io/projected/16498d8f-ea36-4a05-b012-13c320cfe29a-kube-api-access-98c9k\") pod \"16498d8f-ea36-4a05-b012-13c320cfe29a\" (UID: \"16498d8f-ea36-4a05-b012-13c320cfe29a\") "
Oct 30 06:23:50.846584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.846530    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-kube-api-access-mvg2h\" (UniqueName: \"kubernetes.io/projected/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-kube-api-access-mvg2h\") pod \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\" (UID: \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\") "
Oct 30 06:23:50.846637 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.846621    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16498d8f-ea36-4a05-b012-13c320cfe29a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16498d8f-ea36-4a05-b012-13c320cfe29a-plugins-config-volume\") pod \"16498d8f-ea36-4a05-b012-13c320cfe29a\" (UID: \"16498d8f-ea36-4a05-b012-13c320cfe29a\") "
Oct 30 06:23:50.846704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.846690    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-plugins-config-volume\") pod \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\" (UID: \"d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a\") "
Oct 30 06:23:50.847027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.847002    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16498d8f-ea36-4a05-b012-13c320cfe29a-plugins-config-volume" pod "16498d8f-ea36-4a05-b012-13c320cfe29a" (UID: "16498d8f-ea36-4a05-b012-13c320cfe29a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:50.847027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.847016    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-plugins-config-volume" pod "d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a" (UID: "d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:50.849429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.849406    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16498d8f-ea36-4a05-b012-13c320cfe29a-kube-api-access-98c9k" pod "16498d8f-ea36-4a05-b012-13c320cfe29a" (UID: "16498d8f-ea36-4a05-b012-13c320cfe29a"). InnerVolumeSpecName "kube-api-access-98c9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:50.849610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.849596    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-kube-api-access-mvg2h" pod "d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a" (UID: "d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a"). InnerVolumeSpecName "kube-api-access-mvg2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:50.914554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.914529    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:50.948388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.948359    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98c9k\" (UniqueName: \"kubernetes.io/projected/16498d8f-ea36-4a05-b012-13c320cfe29a-kube-api-access-98c9k\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:50.948388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.948387    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvg2h\" (UniqueName: \"kubernetes.io/projected/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-kube-api-access-mvg2h\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:50.948556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.948395    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16498d8f-ea36-4a05-b012-13c320cfe29a-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:50.948556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.948406    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:50.989809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:50.989766    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5" podUID="c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:23:51.154297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.154263    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"]
Oct 30 06:23:51.160163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.160135    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95498c-65544f6767-5dx7b"]
Oct 30 06:23:51.181419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.181391    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:51.188565 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.188543    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"]
Oct 30 06:23:51.193291 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.193268    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95498c-59c99b87b6-ssrst"]
Oct 30 06:23:51.214982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.214958    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"]
Oct 30 06:23:51.218279 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:23:51.218216    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cfef7c_28d9_4fe3_991d_b129e0202ceb.slice/crio-eefb5b79ab883bbb513696228cb896afc1d38dcb0e1c5a7d26c727f0d4112ad4 WatchSource:0}: Error finding container eefb5b79ab883bbb513696228cb896afc1d38dcb0e1c5a7d26c727f0d4112ad4: Status 404 returned error can't find the container with id eefb5b79ab883bbb513696228cb896afc1d38dcb0e1c5a7d26c727f0d4112ad4
Oct 30 06:23:51.281073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.281045    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16498d8f-ea36-4a05-b012-13c320cfe29a" path="/var/lib/kubelet/pods/16498d8f-ea36-4a05-b012-13c320cfe29a/volumes"
Oct 30 06:23:51.281499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.281485    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a" path="/var/lib/kubelet/pods/d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a/volumes"
Oct 30 06:23:51.499656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.499628    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:51.633594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.633558    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:51.780450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.780351    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"]
Oct 30 06:23:51.780640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.780608    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy" containerID="cri-o://8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b" gracePeriod=30
Oct 30 06:23:51.823285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.823243    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw" event={"ID":"c9cfef7c-28d9-4fe3-991d-b129e0202ceb","Type":"ContainerStarted","Data":"10abd841d9faeedf630985336568470c75ec5146c65e96adf76dbaf11b635237"}
Oct 30 06:23:51.823498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.823318    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw" event={"ID":"c9cfef7c-28d9-4fe3-991d-b129e0202ceb","Type":"ContainerStarted","Data":"eefb5b79ab883bbb513696228cb896afc1d38dcb0e1c5a7d26c727f0d4112ad4"}
Oct 30 06:23:51.858419 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.858388    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:51.860529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:51.860485    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw" podStartSLOduration=1.860473726 podStartE2EDuration="1.860473726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:51.858457893 +0000 UTC m=+3146.689495107" watchObservedRunningTime="2026-10-30 06:23:51.860473726 +0000 UTC m=+3146.691510933"
Oct 30 06:23:52.323781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.323749    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"]
Oct 30 06:23:52.324228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.324215    5250 cpu_manager.go:414] "containerMap: removing container" podUID="16498d8f-ea36-4a05-b012-13c320cfe29a" containerName="payload-pre-processing"
Oct 30 06:23:52.324329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.324229    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16498d8f-ea36-4a05-b012-13c320cfe29a" containerName="payload-pre-processing"
Oct 30 06:23:52.324329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.324264    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a" containerName="payload-processing"
Oct 30 06:23:52.324329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.324269    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a" containerName="payload-processing"
Oct 30 06:23:52.324447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.324350    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="16498d8f-ea36-4a05-b012-13c320cfe29a" containerName="payload-pre-processing"
Oct 30 06:23:52.324447 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.324372    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ddf29f-f574-4cb0-8d21-06a4e12f6d4a" containerName="payload-processing"
Oct 30 06:23:52.327546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.327526    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.345831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.345808    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"]
Oct 30 06:23:52.372231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.372199    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjb7\" (UniqueName: \"kubernetes.io/projected/889bd632-6cf8-46bb-a264-4f9bff494671-kube-api-access-qjjb7\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.372431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.372407    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.372503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.372460    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.473470 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.473399    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjb7\" (UniqueName: \"kubernetes.io/projected/889bd632-6cf8-46bb-a264-4f9bff494671-kube-api-access-qjjb7\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.473613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.473498    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.473613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.473527    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.474262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.474243    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.476074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.476049    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.495274 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.495249    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjb7\" (UniqueName: \"kubernetes.io/projected/889bd632-6cf8-46bb-a264-4f9bff494671-kube-api-access-qjjb7\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.640541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.640507    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:52.831951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.831869    5250 generic.go:356] "Generic (PLEG): container finished" podID="c9cfef7c-28d9-4fe3-991d-b129e0202ceb" containerID="10abd841d9faeedf630985336568470c75ec5146c65e96adf76dbaf11b635237" exitCode=0
Oct 30 06:23:52.832104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.831956    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw" event={"ID":"c9cfef7c-28d9-4fe3-991d-b129e0202ceb","Type":"ContainerDied","Data":"10abd841d9faeedf630985336568470c75ec5146c65e96adf76dbaf11b635237"}
Oct 30 06:23:52.941596 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:52.941397    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"]
Oct 30 06:23:52.944784 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:23:52.944753    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889bd632_6cf8_46bb_a264_4f9bff494671.slice/crio-0ac41d9b60e04dfeb1586831db0c020a82c9d522f39e814473db18bdfe07b50d WatchSource:0}: Error finding container 0ac41d9b60e04dfeb1586831db0c020a82c9d522f39e814473db18bdfe07b50d: Status 404 returned error can't find the container with id 0ac41d9b60e04dfeb1586831db0c020a82c9d522f39e814473db18bdfe07b50d
Oct 30 06:23:53.175776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:53.175735    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:53.716611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:53.716577    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:53.839757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:53.839720    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg" event={"ID":"889bd632-6cf8-46bb-a264-4f9bff494671","Type":"ContainerStarted","Data":"62888048f2c85cac44ca39472e77c71b6ee747721578e6b57ecc48add2485027"}
Oct 30 06:23:53.839949 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:53.839783    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg" event={"ID":"889bd632-6cf8-46bb-a264-4f9bff494671","Type":"ContainerStarted","Data":"0ac41d9b60e04dfeb1586831db0c020a82c9d522f39e814473db18bdfe07b50d"}
Oct 30 06:23:53.875967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:53.875910    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg" podStartSLOduration=1.875892986 podStartE2EDuration="1.875892986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:53.875425643 +0000 UTC m=+3148.706462860" watchObservedRunningTime="2026-10-30 06:23:53.875892986 +0000 UTC m=+3148.706930206"
Oct 30 06:23:53.939902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:53.939865    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:54.004004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.003977    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:54.075888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.075857    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:54.092094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.092067    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-kube-api-access-8wx54\" (UniqueName: \"kubernetes.io/projected/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-kube-api-access-8wx54\") pod \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") "
Oct 30 06:23:54.092277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.092104    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-openshift-ca\") pod \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") "
Oct 30 06:23:54.092277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.092134    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-xks-ca\") pod \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\" (UID: \"c9cfef7c-28d9-4fe3-991d-b129e0202ceb\") "
Oct 30 06:23:54.092477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.092453    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-openshift-ca" pod "c9cfef7c-28d9-4fe3-991d-b129e0202ceb" (UID: "c9cfef7c-28d9-4fe3-991d-b129e0202ceb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:54.094400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.094378    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-kube-api-access-8wx54" pod "c9cfef7c-28d9-4fe3-991d-b129e0202ceb" (UID: "c9cfef7c-28d9-4fe3-991d-b129e0202ceb"). InnerVolumeSpecName "kube-api-access-8wx54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:54.094505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.094410    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-xks-ca" pod "c9cfef7c-28d9-4fe3-991d-b129e0202ceb" (UID: "c9cfef7c-28d9-4fe3-991d-b129e0202ceb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:23:54.194198 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.194150    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wx54\" (UniqueName: \"kubernetes.io/projected/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-kube-api-access-8wx54\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:54.194341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.194208    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:54.194341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.194230    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cfef7c-28d9-4fe3-991d-b129e0202ceb-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:54.466008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.465972    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:54.848381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.848293    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"
Oct 30 06:23:54.848381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.848317    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw" event={"ID":"c9cfef7c-28d9-4fe3-991d-b129e0202ceb","Type":"ContainerDied","Data":"eefb5b79ab883bbb513696228cb896afc1d38dcb0e1c5a7d26c727f0d4112ad4"}
Oct 30 06:23:54.848381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.848373    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eefb5b79ab883bbb513696228cb896afc1d38dcb0e1c5a7d26c727f0d4112ad4"
Oct 30 06:23:54.850146 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.850122    5250 generic.go:356] "Generic (PLEG): container finished" podID="889bd632-6cf8-46bb-a264-4f9bff494671" containerID="62888048f2c85cac44ca39472e77c71b6ee747721578e6b57ecc48add2485027" exitCode=0
Oct 30 06:23:54.850314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.850212    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg" event={"ID":"889bd632-6cf8-46bb-a264-4f9bff494671","Type":"ContainerDied","Data":"62888048f2c85cac44ca39472e77c71b6ee747721578e6b57ecc48add2485027"}
Oct 30 06:23:54.880773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.880745    5250 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:23:54.880888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:54.880795    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:23:55.171623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.171598    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:23:55.306287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306259    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-envoy\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.306461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306294    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-credential-socket\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.306461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306324    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-kube-api-access-9n6ts\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-kube-api-access-9n6ts\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.306461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306391    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-certs\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.306642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306542    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-podinfo\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.306642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306563    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-credential-socket" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:55.306642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306601    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-token\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-token\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.306840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306662    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-socket\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.306840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306728    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb46152-58fd-46bd-9d89-14ba2e28208e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb46152-58fd-46bd-9d89-14ba2e28208e-istiod-ca-cert\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.306840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306752    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-certs" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:55.306840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.306785    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-data\") pod \"bdb46152-58fd-46bd-9d89-14ba2e28208e\" (UID: \"bdb46152-58fd-46bd-9d89-14ba2e28208e\") "
Oct 30 06:23:55.307207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.307139    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-data" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:55.307344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.307294    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.307344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.307316    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.307344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.307330    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.307344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.307332    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb46152-58fd-46bd-9d89-14ba2e28208e-istiod-ca-cert" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:55.307552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.307406    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-socket" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:55.308824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.308800    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-envoy" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:55.308824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.308813    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-kube-api-access-9n6ts" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "kube-api-access-9n6ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:55.309464 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.309446    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-podinfo" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:23:55.309558 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.309461    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-token" pod "bdb46152-58fd-46bd-9d89-14ba2e28208e" (UID: "bdb46152-58fd-46bd-9d89-14ba2e28208e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:55.373420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.373381    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"]
Oct 30 06:23:55.383952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.383925    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"]
Oct 30 06:23:55.394206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.394149    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"]
Oct 30 06:23:55.408089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.408021    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n6ts\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-kube-api-access-9n6ts\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.408089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.408048    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.408089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.408059    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.408089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.408068    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.408089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.408076    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb46152-58fd-46bd-9d89-14ba2e28208e-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.408089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.408084    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb46152-58fd-46bd-9d89-14ba2e28208e-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:55.524748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.524711    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:55.850457 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.850355    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:55.858318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.858292    5250 generic.go:356] "Generic (PLEG): container finished" podID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerID="2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980" exitCode=0
Oct 30 06:23:55.858452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.858337    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"
Oct 30 06:23:55.858452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.858373    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" event={"ID":"bdb46152-58fd-46bd-9d89-14ba2e28208e","Type":"ContainerDied","Data":"2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980"}
Oct 30 06:23:55.858452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.858430    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp" event={"ID":"bdb46152-58fd-46bd-9d89-14ba2e28208e","Type":"ContainerDied","Data":"b44786f30128e6f2a85c3ea1077c9437cb98222fe390565cfa3ff635cc68342d"}
Oct 30 06:23:55.858581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.858459    5250 scope.go:122] "RemoveContainer" containerID="2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980"
Oct 30 06:23:55.869997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.869976    5250 scope.go:122] "RemoveContainer" containerID="2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980"
Oct 30 06:23:55.870295 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:55.870273    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980\": container with ID starting with 2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980 not found: ID does not exist" containerID="2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980"
Oct 30 06:23:55.870387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.870309    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980"} err="failed to get container status \"2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980\": rpc error: code = NotFound desc = could not find container \"2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980\": container with ID starting with 2e50e620dc28231bed636db9cc1f7e4cdddfc72bda37df691b3be9d6eb5fa980 not found: ID does not exist"
Oct 30 06:23:55.912647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.912613    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"]
Oct 30 06:23:55.915514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:55.915483    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-985pp"]
Oct 30 06:23:56.010160 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.010131    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:56.113858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.113794    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-openshift-ca\") pod \"889bd632-6cf8-46bb-a264-4f9bff494671\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") "
Oct 30 06:23:56.113982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.113862    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889bd632-6cf8-46bb-a264-4f9bff494671-kube-api-access-qjjb7\" (UniqueName: \"kubernetes.io/projected/889bd632-6cf8-46bb-a264-4f9bff494671-kube-api-access-qjjb7\") pod \"889bd632-6cf8-46bb-a264-4f9bff494671\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") "
Oct 30 06:23:56.113982 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.113950    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-xks-ca\") pod \"889bd632-6cf8-46bb-a264-4f9bff494671\" (UID: \"889bd632-6cf8-46bb-a264-4f9bff494671\") "
Oct 30 06:23:56.114259 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.114230    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-openshift-ca" pod "889bd632-6cf8-46bb-a264-4f9bff494671" (UID: "889bd632-6cf8-46bb-a264-4f9bff494671"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:56.116111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.116090    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-xks-ca" pod "889bd632-6cf8-46bb-a264-4f9bff494671" (UID: "889bd632-6cf8-46bb-a264-4f9bff494671"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:23:56.116197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.116137    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889bd632-6cf8-46bb-a264-4f9bff494671-kube-api-access-qjjb7" pod "889bd632-6cf8-46bb-a264-4f9bff494671" (UID: "889bd632-6cf8-46bb-a264-4f9bff494671"). InnerVolumeSpecName "kube-api-access-qjjb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:56.215283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.215253    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjb7\" (UniqueName: \"kubernetes.io/projected/889bd632-6cf8-46bb-a264-4f9bff494671-kube-api-access-qjjb7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:56.215405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.215291    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:56.215405 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.215305    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889bd632-6cf8-46bb-a264-4f9bff494671-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:56.335657 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.335629    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 06:23:56.362172 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.362147    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 06:23:56.416448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.416414    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf5f568-1f52-426e-94a4-aece3c315cda-kube-api-access-x765m\" (UniqueName: \"kubernetes.io/projected/fcf5f568-1f52-426e-94a4-aece3c315cda-kube-api-access-x765m\") pod \"fcf5f568-1f52-426e-94a4-aece3c315cda\" (UID: \"fcf5f568-1f52-426e-94a4-aece3c315cda\") "
Oct 30 06:23:56.416665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.416512    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-kube-api-access-s7zp2\" (UniqueName: \"kubernetes.io/projected/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-kube-api-access-s7zp2\") pod \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\" (UID: \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\") "
Oct 30 06:23:56.416665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.416547    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf5f568-1f52-426e-94a4-aece3c315cda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf5f568-1f52-426e-94a4-aece3c315cda-plugins-config-volume\") pod \"fcf5f568-1f52-426e-94a4-aece3c315cda\" (UID: \"fcf5f568-1f52-426e-94a4-aece3c315cda\") "
Oct 30 06:23:56.416665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.416649    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-plugins-config-volume\") pod \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\" (UID: \"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f\") "
Oct 30 06:23:56.417605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.417583    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf5f568-1f52-426e-94a4-aece3c315cda-plugins-config-volume" pod "fcf5f568-1f52-426e-94a4-aece3c315cda" (UID: "fcf5f568-1f52-426e-94a4-aece3c315cda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:56.417605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.417595    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-plugins-config-volume" pod "ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" (UID: "ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:56.419092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.419066    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-kube-api-access-s7zp2" pod "ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" (UID: "ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f"). InnerVolumeSpecName "kube-api-access-s7zp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:56.419230 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.419124    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf5f568-1f52-426e-94a4-aece3c315cda-kube-api-access-x765m" pod "fcf5f568-1f52-426e-94a4-aece3c315cda" (UID: "fcf5f568-1f52-426e-94a4-aece3c315cda"). InnerVolumeSpecName "kube-api-access-x765m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:56.518377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.518344    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:56.518377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.518375    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x765m\" (UniqueName: \"kubernetes.io/projected/fcf5f568-1f52-426e-94a4-aece3c315cda-kube-api-access-x765m\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:56.518518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.518388    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7zp2\" (UniqueName: \"kubernetes.io/projected/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f-kube-api-access-s7zp2\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:56.518518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.518400    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf5f568-1f52-426e-94a4-aece3c315cda-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:56.867149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.867119    5250 generic.go:356] "Generic (PLEG): container finished" podID="fcf5f568-1f52-426e-94a4-aece3c315cda" containerID="5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f" exitCode=0
Oct 30 06:23:56.867651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.867210    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"
Oct 30 06:23:56.867651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.867227    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn" event={"ID":"fcf5f568-1f52-426e-94a4-aece3c315cda","Type":"ContainerDied","Data":"5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f"}
Oct 30 06:23:56.867651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.867304    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn" event={"ID":"fcf5f568-1f52-426e-94a4-aece3c315cda","Type":"ContainerDied","Data":"3384f907a689deb74152afbb3643df954b8a24d82034400cf26941fce561ebdb"}
Oct 30 06:23:56.867651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.867335    5250 scope.go:122] "RemoveContainer" containerID="5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f"
Oct 30 06:23:56.869892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.869873    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"
Oct 30 06:23:56.870006 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.869908    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg" event={"ID":"889bd632-6cf8-46bb-a264-4f9bff494671","Type":"ContainerDied","Data":"0ac41d9b60e04dfeb1586831db0c020a82c9d522f39e814473db18bdfe07b50d"}
Oct 30 06:23:56.870006 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.869959    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ac41d9b60e04dfeb1586831db0c020a82c9d522f39e814473db18bdfe07b50d"
Oct 30 06:23:56.871285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.871261    5250 generic.go:356] "Generic (PLEG): container finished" podID="ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" containerID="12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5" exitCode=0
Oct 30 06:23:56.871368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.871301    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"
Oct 30 06:23:56.871442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.871292    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx" event={"ID":"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f","Type":"ContainerDied","Data":"12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5"}
Oct 30 06:23:56.871442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.871423    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx" event={"ID":"ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f","Type":"ContainerDied","Data":"142bf9b30e908ad1bb1943d5ceaee3d34265e9911380118dc0a4067ef6e89dff"}
Oct 30 06:23:56.897455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.897435    5250 scope.go:122] "RemoveContainer" containerID="5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f"
Oct 30 06:23:56.897857 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:56.897837    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f\": container with ID starting with 5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f not found: ID does not exist" containerID="5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f"
Oct 30 06:23:56.897954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.897865    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f"} err="failed to get container status \"5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f\": rpc error: code = NotFound desc = could not find container \"5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f\": container with ID starting with 5a66382c3c50fd5ba03714deff724fe19fc229f2ff3b783eff36447a744d740f not found: ID does not exist"
Oct 30 06:23:56.897954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.897887    5250 scope.go:122] "RemoveContainer" containerID="12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5"
Oct 30 06:23:56.910751 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.910724    5250 scope.go:122] "RemoveContainer" containerID="12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5"
Oct 30 06:23:56.911171 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:56.911108    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5\": container with ID starting with 12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5 not found: ID does not exist" containerID="12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5"
Oct 30 06:23:56.911171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.911143    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5"} err="failed to get container status \"12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5\": rpc error: code = NotFound desc = could not find container \"12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5\": container with ID starting with 12c682d741c5ee4f1155ec3ea56c2bf17fbad3408f2df0dad1d24592c7ed67a5 not found: ID does not exist"
Oct 30 06:23:56.931943 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.931889    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"]
Oct 30 06:23:56.942260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.942227    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-452a59-5dd849f5f9-d4txn"]
Oct 30 06:23:56.985383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.985349    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"]
Oct 30 06:23:56.990325 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:56.990293    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-452a59-847599d748-b9jpx"]
Oct 30 06:23:57.052478 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.052417    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:23:57.124618 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.124586    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-podinfo\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.124829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.124637    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-token\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-token\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.124829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.124661    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-certs\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.124829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.124701    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-socket\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.124829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.124730    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cc351b-86a2-4fd7-9f17-9b831061c445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cc351b-86a2-4fd7-9f17-9b831061c445-istiod-ca-cert\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.124829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.124771    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-data\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.124829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.124808    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-credential-socket\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.125348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.124836    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-envoy\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.125348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.125006    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-certs" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:57.125549 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.125517    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-socket" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:57.125674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.125647    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-credential-socket" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:57.128752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.125727    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-data" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:57.128752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.125939    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-kube-api-access-pfhct\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-kube-api-access-pfhct\") pod \"46cc351b-86a2-4fd7-9f17-9b831061c445\" (UID: \"46cc351b-86a2-4fd7-9f17-9b831061c445\") "
Oct 30 06:23:57.128752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.127501    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cc351b-86a2-4fd7-9f17-9b831061c445-istiod-ca-cert" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:23:57.129696 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.129668    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-podinfo" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:23:57.140623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.131749    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.140623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.131796    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.140623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.131821    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cc351b-86a2-4fd7-9f17-9b831061c445-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.140623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.131834    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.140623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.131846    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.140623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.135393    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-token" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:57.140623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.140609    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-envoy" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:23:57.142737 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.142574    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-kube-api-access-pfhct" pod "46cc351b-86a2-4fd7-9f17-9b831061c445" (UID: "46cc351b-86a2-4fd7-9f17-9b831061c445"). InnerVolumeSpecName "kube-api-access-pfhct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:23:57.233042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.233008    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfhct\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-kube-api-access-pfhct\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.233042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.233037    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.233042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.233047    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.233321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.233055    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cc351b-86a2-4fd7-9f17-9b831061c445-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:23:57.280366 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.280318    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" path="/var/lib/kubelet/pods/bdb46152-58fd-46bd-9d89-14ba2e28208e/volumes"
Oct 30 06:23:57.280815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.280793    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" path="/var/lib/kubelet/pods/ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f/volumes"
Oct 30 06:23:57.281217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.281198    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf5f568-1f52-426e-94a4-aece3c315cda" path="/var/lib/kubelet/pods/fcf5f568-1f52-426e-94a4-aece3c315cda/volumes"
Oct 30 06:23:57.667727 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.667692    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:57.881250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.881151    5250 generic.go:356] "Generic (PLEG): container finished" podID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerID="8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b" exitCode=0
Oct 30 06:23:57.881250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.881225    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" event={"ID":"46cc351b-86a2-4fd7-9f17-9b831061c445","Type":"ContainerDied","Data":"8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b"}
Oct 30 06:23:57.881250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.881230    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"
Oct 30 06:23:57.881835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.881272    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5" event={"ID":"46cc351b-86a2-4fd7-9f17-9b831061c445","Type":"ContainerDied","Data":"3ccc2886a1a4721c370ba90190617b1041f67105d9f3d76e75a519068658a0dd"}
Oct 30 06:23:57.881835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.881294    5250 scope.go:122] "RemoveContainer" containerID="8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b"
Oct 30 06:23:57.891704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.891686    5250 scope.go:122] "RemoveContainer" containerID="8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b"
Oct 30 06:23:57.892007 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:23:57.891987    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b\": container with ID starting with 8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b not found: ID does not exist" containerID="8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b"
Oct 30 06:23:57.892074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.892019    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b"} err="failed to get container status \"8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b\": rpc error: code = NotFound desc = could not find container \"8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b\": container with ID starting with 8ddb62ea653948b79dc284adc29c06820a34bb3a6e61fd7047a33e8f3351109b not found: ID does not exist"
Oct 30 06:23:57.938655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.938626    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"]
Oct 30 06:23:57.942118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:57.942091    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-zwjr5"]
Oct 30 06:23:59.279830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:59.279797    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" path="/var/lib/kubelet/pods/46cc351b-86a2-4fd7-9f17-9b831061c445/volumes"
Oct 30 06:23:59.288606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:59.288579    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:23:59.977523 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:23:59.977488    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj" podUID="b56bf06b-54bc-4ba4-811b-111b4b02bd0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:24:00.989759 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:00.989725    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5" podUID="c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:24:01.064439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.064410    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:01.209151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.209122    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:01.355020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.354970    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:01.492451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.492412    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:24:01.508487 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.508461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:01.515926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.515905    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:24:01.574258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.573898    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-plugins-config-volume\") pod \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\" (UID: \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\") "
Oct 30 06:24:01.574258 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.574043    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-kube-api-access-zs9v7\" (UniqueName: \"kubernetes.io/projected/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-kube-api-access-zs9v7\") pod \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\" (UID: \"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6\") "
Oct 30 06:24:01.575586 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.575553    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-plugins-config-volume" pod "c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" (UID: "c85a1613-f0cb-4107-9ce4-3ab7b2b220c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:01.580510 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.580462    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-kube-api-access-zs9v7" pod "c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" (UID: "c85a1613-f0cb-4107-9ce4-3ab7b2b220c6"). InnerVolumeSpecName "kube-api-access-zs9v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:01.676363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.676325    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-kube-api-access-9jm58\" (UniqueName: \"kubernetes.io/projected/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-kube-api-access-9jm58\") pod \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\" (UID: \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\") "
Oct 30 06:24:01.676543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.676446    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-plugins-config-volume\") pod \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\" (UID: \"b56bf06b-54bc-4ba4-811b-111b4b02bd0c\") "
Oct 30 06:24:01.676667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.676654    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs9v7\" (UniqueName: \"kubernetes.io/projected/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-kube-api-access-zs9v7\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:01.676708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.676670    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:01.676810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.676790    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-plugins-config-volume" pod "b56bf06b-54bc-4ba4-811b-111b4b02bd0c" (UID: "b56bf06b-54bc-4ba4-811b-111b4b02bd0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:01.678591 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.678571    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-kube-api-access-9jm58" pod "b56bf06b-54bc-4ba4-811b-111b4b02bd0c" (UID: "b56bf06b-54bc-4ba4-811b-111b4b02bd0c"). InnerVolumeSpecName "kube-api-access-9jm58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:01.778240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.778173    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jm58\" (UniqueName: \"kubernetes.io/projected/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-kube-api-access-9jm58\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:01.778240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.778235    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56bf06b-54bc-4ba4-811b-111b4b02bd0c-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:01.904646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.904614    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:01.913105 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.913078    5250 generic.go:356] "Generic (PLEG): container finished" podID="c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" containerID="2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b" exitCode=0
Oct 30 06:24:01.913292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.913130    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"
Oct 30 06:24:01.913292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.913165    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5" event={"ID":"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6","Type":"ContainerDied","Data":"2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b"}
Oct 30 06:24:01.913292 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.913277    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5" event={"ID":"c85a1613-f0cb-4107-9ce4-3ab7b2b220c6","Type":"ContainerDied","Data":"4792310eca80fe3f8b3d085ec3787076f3d26bf481c25128842270531abc44fd"}
Oct 30 06:24:01.913492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.913305    5250 scope.go:122] "RemoveContainer" containerID="2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b"
Oct 30 06:24:01.914808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.914785    5250 generic.go:356] "Generic (PLEG): container finished" podID="b56bf06b-54bc-4ba4-811b-111b4b02bd0c" containerID="66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5" exitCode=0
Oct 30 06:24:01.914929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.914848    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"
Oct 30 06:24:01.914929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.914868    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj" event={"ID":"b56bf06b-54bc-4ba4-811b-111b4b02bd0c","Type":"ContainerDied","Data":"66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5"}
Oct 30 06:24:01.914929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.914923    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj" event={"ID":"b56bf06b-54bc-4ba4-811b-111b4b02bd0c","Type":"ContainerDied","Data":"99f69c8e295be039a8eaacecd51d81ea90d7d9791dd38713a0b760cf5ea5c773"}
Oct 30 06:24:01.924535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.924516    5250 scope.go:122] "RemoveContainer" containerID="2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b"
Oct 30 06:24:01.924843 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:01.924822    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b\": container with ID starting with 2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b not found: ID does not exist" containerID="2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b"
Oct 30 06:24:01.924928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.924850    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b"} err="failed to get container status \"2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b\": rpc error: code = NotFound desc = could not find container \"2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b\": container with ID starting with 2ab77c2b0d69db376448dfe81bc334907cd619e62d6a22b5ebffcf88e900549b not found: ID does not exist"
Oct 30 06:24:01.924928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.924867    5250 scope.go:122] "RemoveContainer" containerID="66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5"
Oct 30 06:24:01.935551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.935533    5250 scope.go:122] "RemoveContainer" containerID="66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5"
Oct 30 06:24:01.935810 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:01.935785    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5\": container with ID starting with 66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5 not found: ID does not exist" containerID="66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5"
Oct 30 06:24:01.935865 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.935809    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5"} err="failed to get container status \"66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5\": rpc error: code = NotFound desc = could not find container \"66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5\": container with ID starting with 66cbb050cf68c25511f0083cc7b96cf40c6c95b3cf2108255691d17188ccd6b5 not found: ID does not exist"
Oct 30 06:24:01.970273 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.970248    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"]
Oct 30 06:24:01.974492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:01.974469    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c2be21-6cc8fb4dd7-rkxpj"]
Oct 30 06:24:02.002873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.002845    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"]
Oct 30 06:24:02.007360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.007339    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c2be21-65798b74d4-g6rz5"]
Oct 30 06:24:02.036479 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036454    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"]
Oct 30 06:24:02.036851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036839    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c9cfef7c-28d9-4fe3-991d-b129e0202ceb" containerName="revoke-keys"
Oct 30 06:24:02.036900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036853    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9cfef7c-28d9-4fe3-991d-b129e0202ceb" containerName="revoke-keys"
Oct 30 06:24:02.036900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036867    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b56bf06b-54bc-4ba4-811b-111b4b02bd0c" containerName="payload-pre-processing"
Oct 30 06:24:02.036900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036873    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56bf06b-54bc-4ba4-811b-111b4b02bd0c" containerName="payload-pre-processing"
Oct 30 06:24:02.036900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036882    5250 cpu_manager.go:414] "containerMap: removing container" podUID="fcf5f568-1f52-426e-94a4-aece3c315cda" containerName="payload-pre-processing"
Oct 30 06:24:02.036900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036887    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf5f568-1f52-426e-94a4-aece3c315cda" containerName="payload-pre-processing"
Oct 30 06:24:02.036900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036901    5250 cpu_manager.go:414] "containerMap: removing container" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036906    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036920    5250 cpu_manager.go:414] "containerMap: removing container" podUID="ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" containerName="payload-processing"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036925    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" containerName="payload-processing"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036939    5250 cpu_manager.go:414] "containerMap: removing container" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036943    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036950    5250 cpu_manager.go:414] "containerMap: removing container" podUID="889bd632-6cf8-46bb-a264-4f9bff494671" containerName="revoke-keys"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036954    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889bd632-6cf8-46bb-a264-4f9bff494671" containerName="revoke-keys"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036959    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" containerName="payload-processing"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.036963    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" containerName="payload-processing"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.037022    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9cfef7c-28d9-4fe3-991d-b129e0202ceb" containerName="revoke-keys"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.037031    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56bf06b-54bc-4ba4-811b-111b4b02bd0c" containerName="payload-pre-processing"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.037036    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf5f568-1f52-426e-94a4-aece3c315cda" containerName="payload-pre-processing"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.037043    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cc351b-86a2-4fd7-9f17-9b831061c445" containerName="istio-proxy"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.037050    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3c1d2d-1c08-4ea9-b74c-426e8ab1238f" containerName="payload-processing"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.037057    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb46152-58fd-46bd-9d89-14ba2e28208e" containerName="istio-proxy"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.037062    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="889bd632-6cf8-46bb-a264-4f9bff494671" containerName="revoke-keys"
Oct 30 06:24:02.037116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.037066    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" containerName="payload-processing"
Oct 30 06:24:02.041816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.041798    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.055673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.055652    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:24:02.055976 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.055949    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:24:02.058790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.058765    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"]
Oct 30 06:24:02.181032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.180945    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.181173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.181080    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnn6h\" (UniqueName: \"kubernetes.io/projected/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-kube-api-access-jnn6h\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.181173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.181118    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.282033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.281998    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.282166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.282098    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnn6h\" (UniqueName: \"kubernetes.io/projected/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-kube-api-access-jnn6h\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.282166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.282136    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.282773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.282756    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.284531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.284510    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.303161 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.303131    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnn6h\" (UniqueName: \"kubernetes.io/projected/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-kube-api-access-jnn6h\") pod \"maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.355047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.355017    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:02.375491 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:02.375460    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:24:02.604218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.604114    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"]
Oct 30 06:24:02.608593 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.608561    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.627116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.627067    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"]
Oct 30 06:24:02.664530 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.664384    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"]
Oct 30 06:24:02.667961 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:24:02.667933    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1592c8c2_f7dc_4c14_84cb_877e5088ceb4.slice/crio-1a2e48b5afe512c1a197a2d188f6041da727551c0cd290ec5f8a94002c95cedc WatchSource:0}: Error finding container 1a2e48b5afe512c1a197a2d188f6041da727551c0cd290ec5f8a94002c95cedc: Status 404 returned error can't find the container with id 1a2e48b5afe512c1a197a2d188f6041da727551c0cd290ec5f8a94002c95cedc
Oct 30 06:24:02.683890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.683866    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"]
Oct 30 06:24:02.787835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.787784    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tst\" (UniqueName: \"kubernetes.io/projected/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-kube-api-access-w9tst\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.788030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.787979    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.788151 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.788129    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.835260 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.835208    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:02.894369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.889011    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tst\" (UniqueName: \"kubernetes.io/projected/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-kube-api-access-w9tst\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.894369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.889233    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.894369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.889396    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.894369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.891536    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.894369 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.894141    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.911894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.911871    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tst\" (UniqueName: \"kubernetes.io/projected/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-kube-api-access-w9tst\") pod \"maas-api-revoke-sub-1b49ce39-qz2fm\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.923342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.923322    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:02.924556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.924531    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5" event={"ID":"1592c8c2-f7dc-4c14-84cb-877e5088ceb4","Type":"ContainerStarted","Data":"1a2e48b5afe512c1a197a2d188f6041da727551c0cd290ec5f8a94002c95cedc"}
Oct 30 06:24:02.989295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:02.984582    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:03.195173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:03.195131    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:03.234220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:03.228645    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"]
Oct 30 06:24:03.238558 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:24:03.238501    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104b7ef7_e9bc_4f7f_8201_76e815ca69f3.slice/crio-92dee4f1a744ee42868bfc551d3ab92a7231f0544321f9811a8bc3fc954cdaa7 WatchSource:0}: Error finding container 92dee4f1a744ee42868bfc551d3ab92a7231f0544321f9811a8bc3fc954cdaa7: Status 404 returned error can't find the container with id 92dee4f1a744ee42868bfc551d3ab92a7231f0544321f9811a8bc3fc954cdaa7
Oct 30 06:24:03.280310 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:03.280280    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56bf06b-54bc-4ba4-811b-111b4b02bd0c" path="/var/lib/kubelet/pods/b56bf06b-54bc-4ba4-811b-111b4b02bd0c/volumes"
Oct 30 06:24:03.280772 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:03.280751    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85a1613-f0cb-4107-9ce4-3ab7b2b220c6" path="/var/lib/kubelet/pods/c85a1613-f0cb-4107-9ce4-3ab7b2b220c6/volumes"
Oct 30 06:24:03.932400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:03.932363    5250 generic.go:356] "Generic (PLEG): container finished" podID="1592c8c2-f7dc-4c14-84cb-877e5088ceb4" containerID="66fd3c5bc8f2cf04b7edc983a9d60d255b3b77744a664ac5dc1696308bc349a2" exitCode=0
Oct 30 06:24:03.932584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:03.932410    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5" event={"ID":"1592c8c2-f7dc-4c14-84cb-877e5088ceb4","Type":"ContainerDied","Data":"66fd3c5bc8f2cf04b7edc983a9d60d255b3b77744a664ac5dc1696308bc349a2"}
Oct 30 06:24:03.934256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:03.934228    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm" event={"ID":"104b7ef7-e9bc-4f7f-8201-76e815ca69f3","Type":"ContainerStarted","Data":"c2d045f2160b1f0071ea4dbfd1f1c7c5e83826ba0d5ead0c664bf5843903e9e5"}
Oct 30 06:24:03.934393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:03.934277    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm" event={"ID":"104b7ef7-e9bc-4f7f-8201-76e815ca69f3","Type":"ContainerStarted","Data":"92dee4f1a744ee42868bfc551d3ab92a7231f0544321f9811a8bc3fc954cdaa7"}
Oct 30 06:24:04.005562 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:04.005506    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm" podStartSLOduration=2.005492769 podStartE2EDuration="2.005492769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:04.003670813 +0000 UTC m=+3158.834708030" watchObservedRunningTime="2026-10-30 06:24:04.005492769 +0000 UTC m=+3158.836529986"
Oct 30 06:24:04.840503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:04.840461    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:04.942961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:04.942925    5250 generic.go:356] "Generic (PLEG): container finished" podID="104b7ef7-e9bc-4f7f-8201-76e815ca69f3" containerID="c2d045f2160b1f0071ea4dbfd1f1c7c5e83826ba0d5ead0c664bf5843903e9e5" exitCode=0
Oct 30 06:24:04.943135 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:04.943001    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm" event={"ID":"104b7ef7-e9bc-4f7f-8201-76e815ca69f3","Type":"ContainerDied","Data":"c2d045f2160b1f0071ea4dbfd1f1c7c5e83826ba0d5ead0c664bf5843903e9e5"}
Oct 30 06:24:05.112773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.112748    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:05.213862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.213823    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-xks-ca\") pod \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") "
Oct 30 06:24:05.214069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.213931    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-openshift-ca\") pod \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") "
Oct 30 06:24:05.214069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.213973    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-kube-api-access-jnn6h\" (UniqueName: \"kubernetes.io/projected/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-kube-api-access-jnn6h\") pod \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\" (UID: \"1592c8c2-f7dc-4c14-84cb-877e5088ceb4\") "
Oct 30 06:24:05.214326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.214300    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-openshift-ca" pod "1592c8c2-f7dc-4c14-84cb-877e5088ceb4" (UID: "1592c8c2-f7dc-4c14-84cb-877e5088ceb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:05.216265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.216236    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-xks-ca" pod "1592c8c2-f7dc-4c14-84cb-877e5088ceb4" (UID: "1592c8c2-f7dc-4c14-84cb-877e5088ceb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:05.216265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.216258    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-kube-api-access-jnn6h" pod "1592c8c2-f7dc-4c14-84cb-877e5088ceb4" (UID: "1592c8c2-f7dc-4c14-84cb-877e5088ceb4"). InnerVolumeSpecName "kube-api-access-jnn6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:05.315246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.315216    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:05.315246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.315239    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnn6h\" (UniqueName: \"kubernetes.io/projected/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-kube-api-access-jnn6h\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:05.315246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.315249    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1592c8c2-f7dc-4c14-84cb-877e5088ceb4-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:05.402267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.402237    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:05.505503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.505467    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"]
Oct 30 06:24:05.507702 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.507675    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2f1e4c-r4zdg"]
Oct 30 06:24:05.602492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.602457    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:05.950890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.950863    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"
Oct 30 06:24:05.951338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.950894    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5" event={"ID":"1592c8c2-f7dc-4c14-84cb-877e5088ceb4","Type":"ContainerDied","Data":"1a2e48b5afe512c1a197a2d188f6041da727551c0cd290ec5f8a94002c95cedc"}
Oct 30 06:24:05.951338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:05.950944    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a2e48b5afe512c1a197a2d188f6041da727551c0cd290ec5f8a94002c95cedc"
Oct 30 06:24:06.014218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.014157    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:06.085047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.085023    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:06.225851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.225773    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-kube-api-access-w9tst\" (UniqueName: \"kubernetes.io/projected/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-kube-api-access-w9tst\") pod \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") "
Oct 30 06:24:06.225851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.225839    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-xks-ca\") pod \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") "
Oct 30 06:24:06.226122 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.225865    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-openshift-ca\") pod \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\" (UID: \"104b7ef7-e9bc-4f7f-8201-76e815ca69f3\") "
Oct 30 06:24:06.226313 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.226283    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-openshift-ca" pod "104b7ef7-e9bc-4f7f-8201-76e815ca69f3" (UID: "104b7ef7-e9bc-4f7f-8201-76e815ca69f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:06.228215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.228162    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-xks-ca" pod "104b7ef7-e9bc-4f7f-8201-76e815ca69f3" (UID: "104b7ef7-e9bc-4f7f-8201-76e815ca69f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:06.228215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.228199    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-kube-api-access-w9tst" pod "104b7ef7-e9bc-4f7f-8201-76e815ca69f3" (UID: "104b7ef7-e9bc-4f7f-8201-76e815ca69f3"). InnerVolumeSpecName "kube-api-access-w9tst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:06.327663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.327633    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9tst\" (UniqueName: \"kubernetes.io/projected/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-kube-api-access-w9tst\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:06.327663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.327661    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:06.327663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.327671    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104b7ef7-e9bc-4f7f-8201-76e815ca69f3-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:06.837542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.837503    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"]
Oct 30 06:24:06.847725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.847695    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"]
Oct 30 06:24:06.959005 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.958971    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"
Oct 30 06:24:06.959520 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.959004    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm" event={"ID":"104b7ef7-e9bc-4f7f-8201-76e815ca69f3","Type":"ContainerDied","Data":"92dee4f1a744ee42868bfc551d3ab92a7231f0544321f9811a8bc3fc954cdaa7"}
Oct 30 06:24:06.959520 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.959049    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92dee4f1a744ee42868bfc551d3ab92a7231f0544321f9811a8bc3fc954cdaa7"
Oct 30 06:24:06.991745 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:06.991709    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:07.280504 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:07.280467    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889bd632-6cf8-46bb-a264-4f9bff494671" path="/var/lib/kubelet/pods/889bd632-6cf8-46bb-a264-4f9bff494671/volumes"
Oct 30 06:24:07.528255 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:07.528212    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:07.684885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:07.684847    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq" podUID="9ce112f1-37d7-4061-bc37-dcab95805834" containerName="payload-processing" containerID="cri-o://0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd" gracePeriod=25
Oct 30 06:24:09.406682 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:09.406645    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:10.395610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.395574    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main" containerID="cri-o://c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6" gracePeriod=45
Oct 30 06:24:10.404967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.404916    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" containerID="cri-o://1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009" gracePeriod=45
Oct 30 06:24:10.414077 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.414038    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main" containerID="cri-o://bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9" gracePeriod=45
Oct 30 06:24:10.432714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.432686    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused"
Oct 30 06:24:10.693997 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.693827    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 06:24:10.697400 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.697376    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 06:24:10.724482 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.724452    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 06:24:10.876277 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876244    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-model-cache\") pod \"2cff40c6-828b-4662-b068-5938a065ef90\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") "
Oct 30 06:24:10.876485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876315    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bacfe6b-fd8d-485c-8f23-63d665188be5-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bacfe6b-fd8d-485c-8f23-63d665188be5-tls-certs\") pod \"4bacfe6b-fd8d-485c-8f23-63d665188be5\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") "
Oct 30 06:24:10.876485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876355    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-dshm\") pod \"4bacfe6b-fd8d-485c-8f23-63d665188be5\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") "
Oct 30 06:24:10.876485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876393    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-model-cache\") pod \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") "
Oct 30 06:24:10.876485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876432    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-home\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-home\") pod \"2cff40c6-828b-4662-b068-5938a065ef90\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") "
Oct 30 06:24:10.876485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876456    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-tmp-dir\") pod \"4bacfe6b-fd8d-485c-8f23-63d665188be5\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") "
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876521    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-tmp-dir\") pod \"2cff40c6-828b-4662-b068-5938a065ef90\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") "
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876549    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-model-cache" pod "2cff40c6-828b-4662-b068-5938a065ef90" (UID: "2cff40c6-828b-4662-b068-5938a065ef90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876577    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-home\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-home\") pod \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") "
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876609    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cff40c6-828b-4662-b068-5938a065ef90-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff40c6-828b-4662-b068-5938a065ef90-tls-certs\") pod \"2cff40c6-828b-4662-b068-5938a065ef90\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") "
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876641    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tmp-dir\") pod \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") "
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876689    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bacfe6b-fd8d-485c-8f23-63d665188be5-kube-api-access-ppcbt\" (UniqueName: \"kubernetes.io/projected/4bacfe6b-fd8d-485c-8f23-63d665188be5-kube-api-access-ppcbt\") pod \"4bacfe6b-fd8d-485c-8f23-63d665188be5\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") "
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876734    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-model-cache" pod "3cab2e63-c729-4d45-9fd7-7748cb5add6b" (UID: "3cab2e63-c729-4d45-9fd7-7748cb5add6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876740    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cab2e63-c729-4d45-9fd7-7748cb5add6b-kube-api-access-sgbns\" (UniqueName: \"kubernetes.io/projected/3cab2e63-c729-4d45-9fd7-7748cb5add6b-kube-api-access-sgbns\") pod \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") "
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876755    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-tmp-dir" pod "4bacfe6b-fd8d-485c-8f23-63d665188be5" (UID: "4bacfe6b-fd8d-485c-8f23-63d665188be5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.876808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876797    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-dshm\") pod \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") "
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876863    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-model-cache\") pod \"4bacfe6b-fd8d-485c-8f23-63d665188be5\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") "
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876917    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-home\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-home\") pod \"4bacfe6b-fd8d-485c-8f23-63d665188be5\" (UID: \"4bacfe6b-fd8d-485c-8f23-63d665188be5\") "
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876952    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-dshm\") pod \"2cff40c6-828b-4662-b068-5938a065ef90\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") "
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.876956    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tmp-dir" pod "3cab2e63-c729-4d45-9fd7-7748cb5add6b" (UID: "3cab2e63-c729-4d45-9fd7-7748cb5add6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877001    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cff40c6-828b-4662-b068-5938a065ef90-kube-api-access-9vw56\" (UniqueName: \"kubernetes.io/projected/2cff40c6-828b-4662-b068-5938a065ef90-kube-api-access-9vw56\") pod \"2cff40c6-828b-4662-b068-5938a065ef90\" (UID: \"2cff40c6-828b-4662-b068-5938a065ef90\") "
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877034    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tls-certs\") pod \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\" (UID: \"3cab2e63-c729-4d45-9fd7-7748cb5add6b\") "
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877140    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-model-cache" pod "4bacfe6b-fd8d-485c-8f23-63d665188be5" (UID: "4bacfe6b-fd8d-485c-8f23-63d665188be5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877148    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-home" pod "3cab2e63-c729-4d45-9fd7-7748cb5add6b" (UID: "3cab2e63-c729-4d45-9fd7-7748cb5add6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.877508 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877510    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.878132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877530    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.878132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877544    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.878132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877559    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.878132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877571    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.878132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877584    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.878132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877723    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-home" pod "2cff40c6-828b-4662-b068-5938a065ef90" (UID: "2cff40c6-828b-4662-b068-5938a065ef90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.878132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.877812    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-tmp-dir" pod "2cff40c6-828b-4662-b068-5938a065ef90" (UID: "2cff40c6-828b-4662-b068-5938a065ef90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.878132 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.878092    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-home" pod "4bacfe6b-fd8d-485c-8f23-63d665188be5" (UID: "4bacfe6b-fd8d-485c-8f23-63d665188be5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.879623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.879593    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cff40c6-828b-4662-b068-5938a065ef90-tls-certs" pod "2cff40c6-828b-4662-b068-5938a065ef90" (UID: "2cff40c6-828b-4662-b068-5938a065ef90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:10.879623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.879592    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-dshm" pod "4bacfe6b-fd8d-485c-8f23-63d665188be5" (UID: "4bacfe6b-fd8d-485c-8f23-63d665188be5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.880683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.880656    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cab2e63-c729-4d45-9fd7-7748cb5add6b-kube-api-access-sgbns" pod "3cab2e63-c729-4d45-9fd7-7748cb5add6b" (UID: "3cab2e63-c729-4d45-9fd7-7748cb5add6b"). InnerVolumeSpecName "kube-api-access-sgbns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:10.880798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.880705    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-dshm" pod "3cab2e63-c729-4d45-9fd7-7748cb5add6b" (UID: "3cab2e63-c729-4d45-9fd7-7748cb5add6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.880863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.880817    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bacfe6b-fd8d-485c-8f23-63d665188be5-tls-certs" pod "4bacfe6b-fd8d-485c-8f23-63d665188be5" (UID: "4bacfe6b-fd8d-485c-8f23-63d665188be5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:10.881048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.881025    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bacfe6b-fd8d-485c-8f23-63d665188be5-kube-api-access-ppcbt" pod "4bacfe6b-fd8d-485c-8f23-63d665188be5" (UID: "4bacfe6b-fd8d-485c-8f23-63d665188be5"). InnerVolumeSpecName "kube-api-access-ppcbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:10.881117 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.881106    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tls-certs" pod "3cab2e63-c729-4d45-9fd7-7748cb5add6b" (UID: "3cab2e63-c729-4d45-9fd7-7748cb5add6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:10.881594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.881571    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-dshm" pod "2cff40c6-828b-4662-b068-5938a065ef90" (UID: "2cff40c6-828b-4662-b068-5938a065ef90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:10.882284 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.882264    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cff40c6-828b-4662-b068-5938a065ef90-kube-api-access-9vw56" pod "2cff40c6-828b-4662-b068-5938a065ef90" (UID: "2cff40c6-828b-4662-b068-5938a065ef90"). InnerVolumeSpecName "kube-api-access-9vw56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:10.978714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978626    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.978714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978657    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff40c6-828b-4662-b068-5938a065ef90-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.978714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978671    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcbt\" (UniqueName: \"kubernetes.io/projected/4bacfe6b-fd8d-485c-8f23-63d665188be5-kube-api-access-ppcbt\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.978714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978685    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbns\" (UniqueName: \"kubernetes.io/projected/3cab2e63-c729-4d45-9fd7-7748cb5add6b-kube-api-access-sgbns\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.978714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978696    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cab2e63-c729-4d45-9fd7-7748cb5add6b-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.978714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978707    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.978714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978718    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.979118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978729    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vw56\" (UniqueName: \"kubernetes.io/projected/2cff40c6-828b-4662-b068-5938a065ef90-kube-api-access-9vw56\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.979118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978740    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cab2e63-c729-4d45-9fd7-7748cb5add6b-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.979118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978751    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bacfe6b-fd8d-485c-8f23-63d665188be5-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.979118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978761    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bacfe6b-fd8d-485c-8f23-63d665188be5-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.979118 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.978770    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff40c6-828b-4662-b068-5938a065ef90-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:10.985898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.985863    5250 generic.go:356] "Generic (PLEG): container finished" podID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerID="c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6" exitCode=0
Oct 30 06:24:10.986058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.985933    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"
Oct 30 06:24:10.986058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.985947    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" event={"ID":"3cab2e63-c729-4d45-9fd7-7748cb5add6b","Type":"ContainerDied","Data":"c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6"}
Oct 30 06:24:10.986058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.985993    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm" event={"ID":"3cab2e63-c729-4d45-9fd7-7748cb5add6b","Type":"ContainerDied","Data":"b1d7bd94428ac73b8cf20b715870c1c3d4db11ec6c088144214589307987b6ca"}
Oct 30 06:24:10.986058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.986011    5250 scope.go:122] "RemoveContainer" containerID="c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6"
Oct 30 06:24:10.987488 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.987463    5250 generic.go:356] "Generic (PLEG): container finished" podID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerID="bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9" exitCode=0
Oct 30 06:24:10.987655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.987549    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" event={"ID":"4bacfe6b-fd8d-485c-8f23-63d665188be5","Type":"ContainerDied","Data":"bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9"}
Oct 30 06:24:10.987655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.987573    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"
Oct 30 06:24:10.988033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.987580    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh" event={"ID":"4bacfe6b-fd8d-485c-8f23-63d665188be5","Type":"ContainerDied","Data":"7ef98b63518220d0b578336be6811b249adf8a89a07e3ab9f88eb472597f7d0c"}
Oct 30 06:24:10.989674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.989650    5250 generic.go:356] "Generic (PLEG): container finished" podID="2cff40c6-828b-4662-b068-5938a065ef90" containerID="1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009" exitCode=0
Oct 30 06:24:10.989780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.989689    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" event={"ID":"2cff40c6-828b-4662-b068-5938a065ef90","Type":"ContainerDied","Data":"1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009"}
Oct 30 06:24:10.989780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.989701    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"
Oct 30 06:24:10.989780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.989715    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8" event={"ID":"2cff40c6-828b-4662-b068-5938a065ef90","Type":"ContainerDied","Data":"dca1dabe8eb9cdc7ef74da559d57c69837026d50d0d81897f2ff02e325c66e4b"}
Oct 30 06:24:10.996871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.996742    5250 scope.go:122] "RemoveContainer" containerID="c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6"
Oct 30 06:24:10.997092 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:10.997069    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6\": container with ID starting with c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6 not found: ID does not exist" containerID="c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6"
Oct 30 06:24:10.997155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.997104    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6"} err="failed to get container status \"c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6\": rpc error: code = NotFound desc = could not find container \"c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6\": container with ID starting with c1dc229e4e41359baa9d45032813bb3918ac40cd53a3bd3ae1540fe35863eac6 not found: ID does not exist"
Oct 30 06:24:10.997155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:10.997140    5250 scope.go:122] "RemoveContainer" containerID="bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9"
Oct 30 06:24:11.008394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.008373    5250 scope.go:122] "RemoveContainer" containerID="bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9"
Oct 30 06:24:11.008716 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:11.008696    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9\": container with ID starting with bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9 not found: ID does not exist" containerID="bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9"
Oct 30 06:24:11.008804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.008724    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9"} err="failed to get container status \"bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9\": rpc error: code = NotFound desc = could not find container \"bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9\": container with ID starting with bc4483d071d84f09cb7ddb1843c7a034a4fd32a80e0972bf00a36ee1e312c0a9 not found: ID does not exist"
Oct 30 06:24:11.008804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.008740    5250 scope.go:122] "RemoveContainer" containerID="1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009"
Oct 30 06:24:11.017511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.017496    5250 scope.go:122] "RemoveContainer" containerID="1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009"
Oct 30 06:24:11.017788 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:11.017772    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009\": container with ID starting with 1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009 not found: ID does not exist" containerID="1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009"
Oct 30 06:24:11.017847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.017793    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009"} err="failed to get container status \"1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009\": rpc error: code = NotFound desc = could not find container \"1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009\": container with ID starting with 1292561c40eb365d677f9687404f941992e622441132979be049fc23a1ca4009 not found: ID does not exist"
Oct 30 06:24:11.058307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.058283    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"]
Oct 30 06:24:11.065103 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.065080    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/simulator-w1-1d8c2f-kserve-6cbfdcfcdd-kvshh"]
Oct 30 06:24:11.094101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.094079    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"]
Oct 30 06:24:11.098355 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.098332    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-w1-1d8c2f-kserve-f98459d48-2gwrm"]
Oct 30 06:24:11.122082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.122059    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:11.129219 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.129167    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"]
Oct 30 06:24:11.133856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.133836    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/premium-w1-1d8c2f-kserve-cf8ddfd9c-kc7x8"]
Oct 30 06:24:11.280359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.280272    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cff40c6-828b-4662-b068-5938a065ef90" path="/var/lib/kubelet/pods/2cff40c6-828b-4662-b068-5938a065ef90/volumes"
Oct 30 06:24:11.280780 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.280762    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" path="/var/lib/kubelet/pods/3cab2e63-c729-4d45-9fd7-7748cb5add6b/volumes"
Oct 30 06:24:11.281246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.281228    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" path="/var/lib/kubelet/pods/4bacfe6b-fd8d-485c-8f23-63d665188be5/volumes"
Oct 30 06:24:11.742540 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:11.742508    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq" podUID="9ce112f1-37d7-4061-bc37-dcab95805834" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:24:12.416366 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:12.416325    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"]
Oct 30 06:24:12.547507 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:12.547472    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:12.682485 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:12.682413    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:12.932404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:12.932368    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:13.140948 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.140912    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:13.277444 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.277408    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:13.682722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.682685    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:13.811064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811030    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"]
Oct 30 06:24:13.811438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811425    5250 cpu_manager.go:414] "containerMap: removing container" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main"
Oct 30 06:24:13.811500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811440    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main"
Oct 30 06:24:13.811500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811456    5250 cpu_manager.go:414] "containerMap: removing container" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main"
Oct 30 06:24:13.811500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811461    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main"
Oct 30 06:24:13.811500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811472    5250 cpu_manager.go:414] "containerMap: removing container" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main"
Oct 30 06:24:13.811500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811477    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main"
Oct 30 06:24:13.811500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811490    5250 cpu_manager.go:414] "containerMap: removing container" podUID="104b7ef7-e9bc-4f7f-8201-76e815ca69f3" containerName="revoke-keys"
Oct 30 06:24:13.811500 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811496    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104b7ef7-e9bc-4f7f-8201-76e815ca69f3" containerName="revoke-keys"
Oct 30 06:24:13.811717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811510    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1592c8c2-f7dc-4c14-84cb-877e5088ceb4" containerName="revoke-keys"
Oct 30 06:24:13.811717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811515    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1592c8c2-f7dc-4c14-84cb-877e5088ceb4" containerName="revoke-keys"
Oct 30 06:24:13.811717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811572    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bacfe6b-fd8d-485c-8f23-63d665188be5" containerName="main"
Oct 30 06:24:13.811717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811583    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cab2e63-c729-4d45-9fd7-7748cb5add6b" containerName="main"
Oct 30 06:24:13.811717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811590    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cff40c6-828b-4662-b068-5938a065ef90" containerName="main"
Oct 30 06:24:13.811717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811596    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="104b7ef7-e9bc-4f7f-8201-76e815ca69f3" containerName="revoke-keys"
Oct 30 06:24:13.811717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.811604    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1592c8c2-f7dc-4c14-84cb-877e5088ceb4" containerName="revoke-keys"
Oct 30 06:24:13.816406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.816388    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:13.833611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.833587    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"]
Oct 30 06:24:13.875130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.875095    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:24:13.875316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.875291    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:24:13.907131 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.907103    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:13.907275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.907156    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:13.907327 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:13.907284    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfc7n\" (UniqueName: \"kubernetes.io/projected/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-kube-api-access-mfc7n\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:14.007797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.007717    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:14.007797 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.007773    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:14.008247 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.007817    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfc7n\" (UniqueName: \"kubernetes.io/projected/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-kube-api-access-mfc7n\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:14.008524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.008506    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:14.010439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.010416    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:14.028933 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.028905    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfc7n\" (UniqueName: \"kubernetes.io/projected/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-kube-api-access-mfc7n\") pod \"maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:14.128887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.128852    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:14.426333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.426301    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"]
Oct 30 06:24:14.429562 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:24:14.429528    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3f2fb6_38fc_4601_af66_eac4ce2c4c30.slice/crio-77f9b2a6c801b2311838fcf23a2f7f9ba1c96682e8c8d8f3b89af8671b73c33c WatchSource:0}: Error finding container 77f9b2a6c801b2311838fcf23a2f7f9ba1c96682e8c8d8f3b89af8671b73c33c: Status 404 returned error can't find the container with id 77f9b2a6c801b2311838fcf23a2f7f9ba1c96682e8c8d8f3b89af8671b73c33c
Oct 30 06:24:14.985950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:14.985910    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:15.020441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:15.020362    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw" event={"ID":"da3f2fb6-38fc-4601-af66-eac4ce2c4c30","Type":"ContainerStarted","Data":"9513b628f964c7951a49262cd738fca76cf188423724259e74207a583abba151"}
Oct 30 06:24:15.020441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:15.020421    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw" event={"ID":"da3f2fb6-38fc-4601-af66-eac4ce2c4c30","Type":"ContainerStarted","Data":"77f9b2a6c801b2311838fcf23a2f7f9ba1c96682e8c8d8f3b89af8671b73c33c"}
Oct 30 06:24:15.060214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:15.060128    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw" podStartSLOduration=2.060115517 podStartE2EDuration="2.060115517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:15.05517294 +0000 UTC m=+3169.886210166" watchObservedRunningTime="2026-10-30 06:24:15.060115517 +0000 UTC m=+3169.891152761"
Oct 30 06:24:16.028680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.028644    5250 generic.go:356] "Generic (PLEG): container finished" podID="da3f2fb6-38fc-4601-af66-eac4ce2c4c30" containerID="9513b628f964c7951a49262cd738fca76cf188423724259e74207a583abba151" exitCode=0
Oct 30 06:24:16.029164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.028733    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw" event={"ID":"da3f2fb6-38fc-4601-af66-eac4ce2c4c30","Type":"ContainerDied","Data":"9513b628f964c7951a49262cd738fca76cf188423724259e74207a583abba151"}
Oct 30 06:24:16.450646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.450609    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"]
Oct 30 06:24:16.454320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.454294    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.473812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.473778    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"]
Oct 30 06:24:16.531901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.531875    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503c6375-f162-4cd9-9748-e3a653399161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.532079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.531933    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trq9\" (UniqueName: \"kubernetes.io/projected/503c6375-f162-4cd9-9748-e3a653399161-kube-api-access-2trq9\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.532079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.531956    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503c6375-f162-4cd9-9748-e3a653399161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.633133 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.633099    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trq9\" (UniqueName: \"kubernetes.io/projected/503c6375-f162-4cd9-9748-e3a653399161-kube-api-access-2trq9\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.633323 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.633139    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503c6375-f162-4cd9-9748-e3a653399161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.633323 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.633249    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503c6375-f162-4cd9-9748-e3a653399161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.633886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.633858    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503c6375-f162-4cd9-9748-e3a653399161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.635837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.635813    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503c6375-f162-4cd9-9748-e3a653399161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.654798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.654772    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trq9\" (UniqueName: \"kubernetes.io/projected/503c6375-f162-4cd9-9748-e3a653399161-kube-api-access-2trq9\") pod \"maas-api-revoke-sub-2e65ec30-xzzvl\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.768374 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.768290    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:16.918363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:16.918322    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:17.070538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.070509    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"]
Oct 30 06:24:17.072555 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:24:17.072519    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503c6375_f162_4cd9_9748_e3a653399161.slice/crio-c674edc3352ff204eef34e56799b3a43b0ed40e06d3e25089e5bb06658d831bd WatchSource:0}: Error finding container c674edc3352ff204eef34e56799b3a43b0ed40e06d3e25089e5bb06658d831bd: Status 404 returned error can't find the container with id c674edc3352ff204eef34e56799b3a43b0ed40e06d3e25089e5bb06658d831bd
Oct 30 06:24:17.153096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.153072    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:17.238352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.238322    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-kube-api-access-mfc7n\" (UniqueName: \"kubernetes.io/projected/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-kube-api-access-mfc7n\") pod \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") "
Oct 30 06:24:17.238566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.238417    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-openshift-ca\") pod \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") "
Oct 30 06:24:17.238566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.238466    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-xks-ca\") pod \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\" (UID: \"da3f2fb6-38fc-4601-af66-eac4ce2c4c30\") "
Oct 30 06:24:17.238784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.238757    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-openshift-ca" pod "da3f2fb6-38fc-4601-af66-eac4ce2c4c30" (UID: "da3f2fb6-38fc-4601-af66-eac4ce2c4c30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:17.240733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.240710    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-kube-api-access-mfc7n" pod "da3f2fb6-38fc-4601-af66-eac4ce2c4c30" (UID: "da3f2fb6-38fc-4601-af66-eac4ce2c4c30"). InnerVolumeSpecName "kube-api-access-mfc7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:17.240831 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.240756    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-xks-ca" pod "da3f2fb6-38fc-4601-af66-eac4ce2c4c30" (UID: "da3f2fb6-38fc-4601-af66-eac4ce2c4c30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:17.339735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.339705    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfc7n\" (UniqueName: \"kubernetes.io/projected/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-kube-api-access-mfc7n\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:17.339735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.339734    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:17.339914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.339744    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f2fb6-38fc-4601-af66-eac4ce2c4c30-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:17.416856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.416823    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4" podUID="9f53076d-3434-4341-bd0c-836f77c610c7" containerName="payload-pre-processing" containerID="cri-o://4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c" gracePeriod=25
Oct 30 06:24:17.679456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.679430    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 06:24:17.744246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.744165    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f53076d-3434-4341-bd0c-836f77c610c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f53076d-3434-4341-bd0c-836f77c610c7-plugins-config-volume\") pod \"9f53076d-3434-4341-bd0c-836f77c610c7\" (UID: \"9f53076d-3434-4341-bd0c-836f77c610c7\") "
Oct 30 06:24:17.744448 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.744261    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f53076d-3434-4341-bd0c-836f77c610c7-kube-api-access-rrztc\" (UniqueName: \"kubernetes.io/projected/9f53076d-3434-4341-bd0c-836f77c610c7-kube-api-access-rrztc\") pod \"9f53076d-3434-4341-bd0c-836f77c610c7\" (UID: \"9f53076d-3434-4341-bd0c-836f77c610c7\") "
Oct 30 06:24:17.744531 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.744493    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f53076d-3434-4341-bd0c-836f77c610c7-plugins-config-volume" pod "9f53076d-3434-4341-bd0c-836f77c610c7" (UID: "9f53076d-3434-4341-bd0c-836f77c610c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:17.744638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.744622    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f53076d-3434-4341-bd0c-836f77c610c7-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:17.746915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.746890    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f53076d-3434-4341-bd0c-836f77c610c7-kube-api-access-rrztc" pod "9f53076d-3434-4341-bd0c-836f77c610c7" (UID: "9f53076d-3434-4341-bd0c-836f77c610c7"). InnerVolumeSpecName "kube-api-access-rrztc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:17.845626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:17.845523    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrztc\" (UniqueName: \"kubernetes.io/projected/9f53076d-3434-4341-bd0c-836f77c610c7-kube-api-access-rrztc\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:18.044892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.044855    5250 generic.go:356] "Generic (PLEG): container finished" podID="503c6375-f162-4cd9-9748-e3a653399161" containerID="16847787e1b2550dff449d52b9038cf468b6fbe4f0bdb86d170bd04dc7116386" exitCode=0
Oct 30 06:24:18.045086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.044938    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl" event={"ID":"503c6375-f162-4cd9-9748-e3a653399161","Type":"ContainerDied","Data":"16847787e1b2550dff449d52b9038cf468b6fbe4f0bdb86d170bd04dc7116386"}
Oct 30 06:24:18.045086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.044994    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl" event={"ID":"503c6375-f162-4cd9-9748-e3a653399161","Type":"ContainerStarted","Data":"c674edc3352ff204eef34e56799b3a43b0ed40e06d3e25089e5bb06658d831bd"}
Oct 30 06:24:18.047451 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.047429    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"
Oct 30 06:24:18.047607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.047466    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw" event={"ID":"da3f2fb6-38fc-4601-af66-eac4ce2c4c30","Type":"ContainerDied","Data":"77f9b2a6c801b2311838fcf23a2f7f9ba1c96682e8c8d8f3b89af8671b73c33c"}
Oct 30 06:24:18.047607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.047505    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77f9b2a6c801b2311838fcf23a2f7f9ba1c96682e8c8d8f3b89af8671b73c33c"
Oct 30 06:24:18.049079 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.049059    5250 generic.go:356] "Generic (PLEG): container finished" podID="9f53076d-3434-4341-bd0c-836f77c610c7" containerID="4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c" exitCode=0
Oct 30 06:24:18.049221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.049095    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4" event={"ID":"9f53076d-3434-4341-bd0c-836f77c610c7","Type":"ContainerDied","Data":"4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c"}
Oct 30 06:24:18.049221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.049108    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"
Oct 30 06:24:18.049221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.049121    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4" event={"ID":"9f53076d-3434-4341-bd0c-836f77c610c7","Type":"ContainerDied","Data":"bd2b4dc1f032b5e2593a0680242f4ecc7ae1bfb171c73454d3a3a799f74a39ec"}
Oct 30 06:24:18.049221 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.049142    5250 scope.go:122] "RemoveContainer" containerID="4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c"
Oct 30 06:24:18.059455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.059436    5250 scope.go:122] "RemoveContainer" containerID="4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c"
Oct 30 06:24:18.059761 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:18.059742    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c\": container with ID starting with 4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c not found: ID does not exist" containerID="4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c"
Oct 30 06:24:18.059832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.059769    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c"} err="failed to get container status \"4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c\": rpc error: code = NotFound desc = could not find container \"4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c\": container with ID starting with 4ece531916076c7445e052647ada345827010f8dfacd83b4fbc9eb6b1988491c not found: ID does not exist"
Oct 30 06:24:18.149094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.149047    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"]
Oct 30 06:24:18.156661 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.156633    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-64123f-77c47fb567-6n4l4"]
Oct 30 06:24:18.589522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.589431    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:18.624602 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.624566    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"]
Oct 30 06:24:18.632693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.632663    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"]
Oct 30 06:24:18.759073 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:18.759032    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:19.185992 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.185969    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:19.259397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.259366    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503c6375-f162-4cd9-9748-e3a653399161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503c6375-f162-4cd9-9748-e3a653399161-maas-api-openshift-ca\") pod \"503c6375-f162-4cd9-9748-e3a653399161\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") "
Oct 30 06:24:19.259566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.259431    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/503c6375-f162-4cd9-9748-e3a653399161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503c6375-f162-4cd9-9748-e3a653399161-maas-api-xks-ca\") pod \"503c6375-f162-4cd9-9748-e3a653399161\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") "
Oct 30 06:24:19.259566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.259462    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503c6375-f162-4cd9-9748-e3a653399161-kube-api-access-2trq9\" (UniqueName: \"kubernetes.io/projected/503c6375-f162-4cd9-9748-e3a653399161-kube-api-access-2trq9\") pod \"503c6375-f162-4cd9-9748-e3a653399161\" (UID: \"503c6375-f162-4cd9-9748-e3a653399161\") "
Oct 30 06:24:19.259805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.259778    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503c6375-f162-4cd9-9748-e3a653399161-maas-api-openshift-ca" pod "503c6375-f162-4cd9-9748-e3a653399161" (UID: "503c6375-f162-4cd9-9748-e3a653399161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:19.261799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.261774    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/503c6375-f162-4cd9-9748-e3a653399161-maas-api-xks-ca" pod "503c6375-f162-4cd9-9748-e3a653399161" (UID: "503c6375-f162-4cd9-9748-e3a653399161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:19.261934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.261842    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503c6375-f162-4cd9-9748-e3a653399161-kube-api-access-2trq9" pod "503c6375-f162-4cd9-9748-e3a653399161" (UID: "503c6375-f162-4cd9-9748-e3a653399161"). InnerVolumeSpecName "kube-api-access-2trq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:19.278985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.278958    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f53076d-3434-4341-bd0c-836f77c610c7" path="/var/lib/kubelet/pods/9f53076d-3434-4341-bd0c-836f77c610c7/volumes"
Oct 30 06:24:19.285960 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.285932    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:19.361021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.360946    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503c6375-f162-4cd9-9748-e3a653399161-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:19.361021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.360971    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trq9\" (UniqueName: \"kubernetes.io/projected/503c6375-f162-4cd9-9748-e3a653399161-kube-api-access-2trq9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:19.361021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.360981    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503c6375-f162-4cd9-9748-e3a653399161-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:19.383222 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.383169    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"]
Oct 30 06:24:19.391301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.391263    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1d8c2f-d2zqw"]
Oct 30 06:24:19.434555 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.434512    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:19.912067 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:19.912021    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:20.065083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:20.065044    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"
Oct 30 06:24:20.065083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:20.065064    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl" event={"ID":"503c6375-f162-4cd9-9748-e3a653399161","Type":"ContainerDied","Data":"c674edc3352ff204eef34e56799b3a43b0ed40e06d3e25089e5bb06658d831bd"}
Oct 30 06:24:20.065317 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:20.065141    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c674edc3352ff204eef34e56799b3a43b0ed40e06d3e25089e5bb06658d831bd"
Oct 30 06:24:21.279815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:21.279785    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9cfef7c-28d9-4fe3-991d-b129e0202ceb" path="/var/lib/kubelet/pods/c9cfef7c-28d9-4fe3-991d-b129e0202ceb/volumes"
Oct 30 06:24:21.429019 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:21.428968    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:21.742635 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:21.742602    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq" podUID="9ce112f1-37d7-4061-bc37-dcab95805834" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:24:21.859514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:21.859472    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" containerID="cri-o://c128d3ff292d0607c296acf33292c6198b870195f7d082d4d8c13d9402b0b6ac" gracePeriod=45
Oct 30 06:24:21.871595 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:21.871567    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" containerID="cri-o://44daaf2ae28f59dd469cf887717554a3756c598f75cf97f82786394779cc1f40" gracePeriod=45
Oct 30 06:24:21.930384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:21.930352    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 06:24:22.081517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.081478    5250 generic.go:356] "Generic (PLEG): container finished" podID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerID="44daaf2ae28f59dd469cf887717554a3756c598f75cf97f82786394779cc1f40" exitCode=0
Oct 30 06:24:22.081679 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.081539    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" event={"ID":"c57dc45f-74b8-4009-a354-2c8d32d570ce","Type":"ContainerDied","Data":"44daaf2ae28f59dd469cf887717554a3756c598f75cf97f82786394779cc1f40"}
Oct 30 06:24:22.083763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.083740    5250 generic.go:356] "Generic (PLEG): container finished" podID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerID="c128d3ff292d0607c296acf33292c6198b870195f7d082d4d8c13d9402b0b6ac" exitCode=0
Oct 30 06:24:22.083916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.083800    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" event={"ID":"573ce1fa-8d94-419b-99e4-ea42aa8ce51a","Type":"ContainerDied","Data":"c128d3ff292d0607c296acf33292c6198b870195f7d082d4d8c13d9402b0b6ac"}
Oct 30 06:24:22.131352 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.131329    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 06:24:22.145600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.145576    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 06:24:22.190728 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.190690    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-model-cache\") pod \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") "
Oct 30 06:24:22.190890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.190743    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tmp-dir\") pod \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") "
Oct 30 06:24:22.190890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.190819    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-dshm\") pod \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") "
Oct 30 06:24:22.190990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.190886    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-tmp-dir\") pod \"c57dc45f-74b8-4009-a354-2c8d32d570ce\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") "
Oct 30 06:24:22.190990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.190941    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-kube-api-access-p8r7p\" (UniqueName: \"kubernetes.io/projected/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-kube-api-access-p8r7p\") pod \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") "
Oct 30 06:24:22.190990 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.190983    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-model-cache\") pod \"c57dc45f-74b8-4009-a354-2c8d32d570ce\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") "
Oct 30 06:24:22.191170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191020    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tls-certs\" (UniqueName: \"kubernetes.io/secret/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tls-certs\") pod \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") "
Oct 30 06:24:22.191170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191018    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-model-cache" pod "573ce1fa-8d94-419b-99e4-ea42aa8ce51a" (UID: "573ce1fa-8d94-419b-99e4-ea42aa8ce51a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:22.191170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191061    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-home\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-home\") pod \"c57dc45f-74b8-4009-a354-2c8d32d570ce\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") "
Oct 30 06:24:22.191170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191092    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57dc45f-74b8-4009-a354-2c8d32d570ce-kube-api-access-dx48s\" (UniqueName: \"kubernetes.io/projected/c57dc45f-74b8-4009-a354-2c8d32d570ce-kube-api-access-dx48s\") pod \"c57dc45f-74b8-4009-a354-2c8d32d570ce\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") "
Oct 30 06:24:22.191170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191094    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tmp-dir" pod "573ce1fa-8d94-419b-99e4-ea42aa8ce51a" (UID: "573ce1fa-8d94-419b-99e4-ea42aa8ce51a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:22.191170 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191137    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57dc45f-74b8-4009-a354-2c8d32d570ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/c57dc45f-74b8-4009-a354-2c8d32d570ce-tls-certs\") pod \"c57dc45f-74b8-4009-a354-2c8d32d570ce\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") "
Oct 30 06:24:22.191606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191247    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-home\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-home\") pod \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\" (UID: \"573ce1fa-8d94-419b-99e4-ea42aa8ce51a\") "
Oct 30 06:24:22.191606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191296    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-dshm\") pod \"c57dc45f-74b8-4009-a354-2c8d32d570ce\" (UID: \"c57dc45f-74b8-4009-a354-2c8d32d570ce\") "
Oct 30 06:24:22.191806 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191782    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.191901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191819    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.191901 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.191785    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-tmp-dir" pod "c57dc45f-74b8-4009-a354-2c8d32d570ce" (UID: "c57dc45f-74b8-4009-a354-2c8d32d570ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:22.192015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.192001    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-model-cache" pod "c57dc45f-74b8-4009-a354-2c8d32d570ce" (UID: "c57dc45f-74b8-4009-a354-2c8d32d570ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:22.192082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.192062    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-home" pod "c57dc45f-74b8-4009-a354-2c8d32d570ce" (UID: "c57dc45f-74b8-4009-a354-2c8d32d570ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:22.192303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.192254    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-home" pod "573ce1fa-8d94-419b-99e4-ea42aa8ce51a" (UID: "573ce1fa-8d94-419b-99e4-ea42aa8ce51a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:22.194689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.194660    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57dc45f-74b8-4009-a354-2c8d32d570ce-kube-api-access-dx48s" pod "c57dc45f-74b8-4009-a354-2c8d32d570ce" (UID: "c57dc45f-74b8-4009-a354-2c8d32d570ce"). InnerVolumeSpecName "kube-api-access-dx48s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:22.194999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.194954    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tls-certs" pod "573ce1fa-8d94-419b-99e4-ea42aa8ce51a" (UID: "573ce1fa-8d94-419b-99e4-ea42aa8ce51a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:22.194999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.194962    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-dshm" pod "573ce1fa-8d94-419b-99e4-ea42aa8ce51a" (UID: "573ce1fa-8d94-419b-99e4-ea42aa8ce51a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:22.194999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.194980    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-dshm" pod "c57dc45f-74b8-4009-a354-2c8d32d570ce" (UID: "c57dc45f-74b8-4009-a354-2c8d32d570ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:22.195303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.195143    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57dc45f-74b8-4009-a354-2c8d32d570ce-tls-certs" pod "c57dc45f-74b8-4009-a354-2c8d32d570ce" (UID: "c57dc45f-74b8-4009-a354-2c8d32d570ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:22.196204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.196154    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-kube-api-access-p8r7p" pod "573ce1fa-8d94-419b-99e4-ea42aa8ce51a" (UID: "573ce1fa-8d94-419b-99e4-ea42aa8ce51a"). InnerVolumeSpecName "kube-api-access-p8r7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:22.292950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292873    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8r7p\" (UniqueName: \"kubernetes.io/projected/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-kube-api-access-p8r7p\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.292950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292901    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.292950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292911    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.292950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292918    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.292950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292927    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx48s\" (UniqueName: \"kubernetes.io/projected/c57dc45f-74b8-4009-a354-2c8d32d570ce-kube-api-access-dx48s\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.292950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292935    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57dc45f-74b8-4009-a354-2c8d32d570ce-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.292950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292944    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.292950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292952    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.293756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292959    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573ce1fa-8d94-419b-99e4-ea42aa8ce51a-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:22.293756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:22.292968    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57dc45f-74b8-4009-a354-2c8d32d570ce-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:23.035853 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.035813    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:24:23.092080 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.092051    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"
Oct 30 06:24:23.092267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.092094    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl" event={"ID":"573ce1fa-8d94-419b-99e4-ea42aa8ce51a","Type":"ContainerDied","Data":"e8912529d4bc80635ddc913f5fc03571901bc5d0aef949cf5020363dba2c23e2"}
Oct 30 06:24:23.092267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.092156    5250 scope.go:122] "RemoveContainer" containerID="c128d3ff292d0607c296acf33292c6198b870195f7d082d4d8c13d9402b0b6ac"
Oct 30 06:24:23.093663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.093649    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"
Oct 30 06:24:23.093788 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.093680    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw" event={"ID":"c57dc45f-74b8-4009-a354-2c8d32d570ce","Type":"ContainerDied","Data":"2d29368968e7386ae10e126a9590840928581091b36dc9a41727dfee9fbcde9f"}
Oct 30 06:24:23.104052 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.104022    5250 scope.go:122] "RemoveContainer" containerID="44daaf2ae28f59dd469cf887717554a3756c598f75cf97f82786394779cc1f40"
Oct 30 06:24:23.151420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.151396    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"]
Oct 30 06:24:23.161127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.161103    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/premium-w6-64123f-kserve-866798d69b-pckkw"]
Oct 30 06:24:23.179171 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.179150    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:23.184533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.184453    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"]
Oct 30 06:24:23.193640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.193604    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-2-w6-64123f-kserve-759d4cbb68-4vmwl"]
Oct 30 06:24:23.279490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.279458    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" path="/var/lib/kubelet/pods/573ce1fa-8d94-419b-99e4-ea42aa8ce51a/volumes"
Oct 30 06:24:23.279871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:23.279859    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" path="/var/lib/kubelet/pods/c57dc45f-74b8-4009-a354-2c8d32d570ce/volumes"
Oct 30 06:24:24.208294 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.208252    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"]
Oct 30 06:24:24.337358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.337323    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:24.480328 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.480244    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:24.574158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574124    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"]
Oct 30 06:24:24.574541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574527    5250 cpu_manager.go:414] "containerMap: removing container" podUID="503c6375-f162-4cd9-9748-e3a653399161" containerName="revoke-keys"
Oct 30 06:24:24.574589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574543    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503c6375-f162-4cd9-9748-e3a653399161" containerName="revoke-keys"
Oct 30 06:24:24.574589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574564    5250 cpu_manager.go:414] "containerMap: removing container" podUID="da3f2fb6-38fc-4601-af66-eac4ce2c4c30" containerName="revoke-keys"
Oct 30 06:24:24.574589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574569    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3f2fb6-38fc-4601-af66-eac4ce2c4c30" containerName="revoke-keys"
Oct 30 06:24:24.574589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574579    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9f53076d-3434-4341-bd0c-836f77c610c7" containerName="payload-pre-processing"
Oct 30 06:24:24.574589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574585    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f53076d-3434-4341-bd0c-836f77c610c7" containerName="payload-pre-processing"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574593    5250 cpu_manager.go:414] "containerMap: removing container" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574597    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574605    5250 cpu_manager.go:414] "containerMap: removing container" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574609    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574668    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3f2fb6-38fc-4601-af66-eac4ce2c4c30" containerName="revoke-keys"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574676    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f53076d-3434-4341-bd0c-836f77c610c7" containerName="payload-pre-processing"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574682    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57dc45f-74b8-4009-a354-2c8d32d570ce" containerName="main"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574689    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="573ce1fa-8d94-419b-99e4-ea42aa8ce51a" containerName="main"
Oct 30 06:24:24.574761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.574697    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="503c6375-f162-4cd9-9748-e3a653399161" containerName="revoke-keys"
Oct 30 06:24:24.579410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.579395    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.593231 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.593204    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:24:24.593420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.593399    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:24:24.596699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.596673    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"]
Oct 30 06:24:24.615018 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.614965    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rb4\" (UniqueName: \"kubernetes.io/projected/311098d1-3cd9-476d-9f9c-3b57a69da5a6-kube-api-access-v4rb4\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.615227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.615023    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.615227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.615066    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.716288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.716257    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.716491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.716301    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.716491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.716452    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rb4\" (UniqueName: \"kubernetes.io/projected/311098d1-3cd9-476d-9f9c-3b57a69da5a6-kube-api-access-v4rb4\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.717061 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.717041    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.718863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.718840    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.743658 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.743586    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rb4\" (UniqueName: \"kubernetes.io/projected/311098d1-3cd9-476d-9f9c-3b57a69da5a6-kube-api-access-v4rb4\") pod \"maas-api-revoke-sub-0e382c51-m2kxs\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:24.893542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:24.893512    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:25.194659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:25.194602    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"]
Oct 30 06:24:25.199070 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:24:25.199041    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311098d1_3cd9_476d_9f9c_3b57a69da5a6.slice/crio-ba264d8050d76b5231de6a40a8bc0adcd5a500db20841e4cfd6bc212a757c245 WatchSource:0}: Error finding container ba264d8050d76b5231de6a40a8bc0adcd5a500db20841e4cfd6bc212a757c245: Status 404 returned error can't find the container with id ba264d8050d76b5231de6a40a8bc0adcd5a500db20841e4cfd6bc212a757c245
Oct 30 06:24:26.117424 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:26.117390    5250 generic.go:356] "Generic (PLEG): container finished" podID="311098d1-3cd9-476d-9f9c-3b57a69da5a6" containerID="5b481b04c3e6418043ef7e4fd8213d204b2e4073c14fb9c25f037d347223bdaa" exitCode=0
Oct 30 06:24:26.117811 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:26.117465    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs" event={"ID":"311098d1-3cd9-476d-9f9c-3b57a69da5a6","Type":"ContainerDied","Data":"5b481b04c3e6418043ef7e4fd8213d204b2e4073c14fb9c25f037d347223bdaa"}
Oct 30 06:24:26.117811 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:26.117517    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs" event={"ID":"311098d1-3cd9-476d-9f9c-3b57a69da5a6","Type":"ContainerStarted","Data":"ba264d8050d76b5231de6a40a8bc0adcd5a500db20841e4cfd6bc212a757c245"}
Oct 30 06:24:26.245075 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:26.245039    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:26.392713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:26.392677    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:26.465245 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:26.465136    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"]
Oct 30 06:24:26.465446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:26.465397    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy" containerID="cri-o://f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928" gracePeriod=30
Oct 30 06:24:26.559849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:26.559793    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:27.266047 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.266019    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:27.348522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.348485    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-xks-ca\") pod \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") "
Oct 30 06:24:27.348717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.348547    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311098d1-3cd9-476d-9f9c-3b57a69da5a6-kube-api-access-v4rb4\" (UniqueName: \"kubernetes.io/projected/311098d1-3cd9-476d-9f9c-3b57a69da5a6-kube-api-access-v4rb4\") pod \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") "
Oct 30 06:24:27.348717 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.348579    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-openshift-ca\") pod \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\" (UID: \"311098d1-3cd9-476d-9f9c-3b57a69da5a6\") "
Oct 30 06:24:27.348991 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.348969    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-openshift-ca" pod "311098d1-3cd9-476d-9f9c-3b57a69da5a6" (UID: "311098d1-3cd9-476d-9f9c-3b57a69da5a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:27.350869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.350844    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311098d1-3cd9-476d-9f9c-3b57a69da5a6-kube-api-access-v4rb4" pod "311098d1-3cd9-476d-9f9c-3b57a69da5a6" (UID: "311098d1-3cd9-476d-9f9c-3b57a69da5a6"). InnerVolumeSpecName "kube-api-access-v4rb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:27.350981 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.350888    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-xks-ca" pod "311098d1-3cd9-476d-9f9c-3b57a69da5a6" (UID: "311098d1-3cd9-476d-9f9c-3b57a69da5a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:27.449592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.449565    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:27.449592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.449589    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4rb4\" (UniqueName: \"kubernetes.io/projected/311098d1-3cd9-476d-9f9c-3b57a69da5a6-kube-api-access-v4rb4\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:27.449771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.449600    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311098d1-3cd9-476d-9f9c-3b57a69da5a6-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:27.917559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:27.917534    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:24:28.055261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.055173    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce112f1-37d7-4061-bc37-dcab95805834-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce112f1-37d7-4061-bc37-dcab95805834-plugins-config-volume\") pod \"9ce112f1-37d7-4061-bc37-dcab95805834\" (UID: \"9ce112f1-37d7-4061-bc37-dcab95805834\") "
Oct 30 06:24:28.055416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.055323    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce112f1-37d7-4061-bc37-dcab95805834-kube-api-access-s54wc\" (UniqueName: \"kubernetes.io/projected/9ce112f1-37d7-4061-bc37-dcab95805834-kube-api-access-s54wc\") pod \"9ce112f1-37d7-4061-bc37-dcab95805834\" (UID: \"9ce112f1-37d7-4061-bc37-dcab95805834\") "
Oct 30 06:24:28.055610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.055584    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce112f1-37d7-4061-bc37-dcab95805834-plugins-config-volume" pod "9ce112f1-37d7-4061-bc37-dcab95805834" (UID: "9ce112f1-37d7-4061-bc37-dcab95805834"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:28.057666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.057637    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce112f1-37d7-4061-bc37-dcab95805834-kube-api-access-s54wc" pod "9ce112f1-37d7-4061-bc37-dcab95805834" (UID: "9ce112f1-37d7-4061-bc37-dcab95805834"). InnerVolumeSpecName "kube-api-access-s54wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:28.131096 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.131070    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"
Oct 30 06:24:28.131280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.131113    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs" event={"ID":"311098d1-3cd9-476d-9f9c-3b57a69da5a6","Type":"ContainerDied","Data":"ba264d8050d76b5231de6a40a8bc0adcd5a500db20841e4cfd6bc212a757c245"}
Oct 30 06:24:28.131280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.131214    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba264d8050d76b5231de6a40a8bc0adcd5a500db20841e4cfd6bc212a757c245"
Oct 30 06:24:28.132647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.132625    5250 generic.go:356] "Generic (PLEG): container finished" podID="9ce112f1-37d7-4061-bc37-dcab95805834" containerID="0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd" exitCode=0
Oct 30 06:24:28.132793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.132677    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"
Oct 30 06:24:28.132793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.132712    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq" event={"ID":"9ce112f1-37d7-4061-bc37-dcab95805834","Type":"ContainerDied","Data":"0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd"}
Oct 30 06:24:28.132793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.132772    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq" event={"ID":"9ce112f1-37d7-4061-bc37-dcab95805834","Type":"ContainerDied","Data":"b3a716cc888663efc60ddf9ca8534514b684648a6d80d56ba7235336470ffdbb"}
Oct 30 06:24:28.132983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.132802    5250 scope.go:122] "RemoveContainer" containerID="0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd"
Oct 30 06:24:28.146167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.146146    5250 scope.go:122] "RemoveContainer" containerID="0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd"
Oct 30 06:24:28.146494 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:28.146475    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd\": container with ID starting with 0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd not found: ID does not exist" containerID="0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd"
Oct 30 06:24:28.146570 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.146504    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd"} err="failed to get container status \"0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd\": rpc error: code = NotFound desc = could not find container \"0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd\": container with ID starting with 0dfb31fbc647c82bc6968d1ad30d4b5220d8ffcb56f6df6c6277912a6d18bfbd not found: ID does not exist"
Oct 30 06:24:28.156554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.156534    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce112f1-37d7-4061-bc37-dcab95805834-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:28.156554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.156555    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s54wc\" (UniqueName: \"kubernetes.io/projected/9ce112f1-37d7-4061-bc37-dcab95805834-kube-api-access-s54wc\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:28.189856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.189831    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"]
Oct 30 06:24:28.194090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:28.194071    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2f1e4c-5649547886-8z5fq"]
Oct 30 06:24:29.037357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:29.037320    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"]
Oct 30 06:24:29.046794 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:29.046767    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2f94e0-lw8lr"]
Oct 30 06:24:29.209521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:29.209454    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8" podUID="f40cfd76-bd78-423e-8ad7-ef3fee34eb52" containerName="payload-pre-processing" containerID="cri-o://6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2" gracePeriod=25
Oct 30 06:24:29.279045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:29.279013    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce112f1-37d7-4061-bc37-dcab95805834" path="/var/lib/kubelet/pods/9ce112f1-37d7-4061-bc37-dcab95805834/volumes"
Oct 30 06:24:29.279452 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:29.279438    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9722ef4-cade-41cb-a8e1-a70ca066bb35" path="/var/lib/kubelet/pods/e9722ef4-cade-41cb-a8e1-a70ca066bb35/volumes"
Oct 30 06:24:29.635044 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:29.635010    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:30.129263 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:30.129214    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:30.228070 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:30.228037    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"]
Oct 30 06:24:30.239372 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:30.239337    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-64123f-l45q5"]
Oct 30 06:24:30.270566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:30.270540    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:30.765186 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:30.765156    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:31.279721 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.279685    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1592c8c2-f7dc-4c14-84cb-877e5088ceb4" path="/var/lib/kubelet/pods/1592c8c2-f7dc-4c14-84cb-877e5088ceb4/volumes"
Oct 30 06:24:31.718417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.718395    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 06:24:31.791351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791317    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-certs\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.791528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791371    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-envoy\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.791528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791408    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-token\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.791528 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791444    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-credential-socket\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.791686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791530    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-data\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.791686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791571    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-kube-api-access-fm2fs\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-kube-api-access-fm2fs\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.791686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791611    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-socket\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.791686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791646    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-podinfo\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.791686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791660    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-certs" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:31.791686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791676    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istiod-ca-cert\") pod \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\" (UID: \"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8\") "
Oct 30 06:24:31.792016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791863    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-credential-socket" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:31.792088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.791874    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-socket" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:31.792157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.792120    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:31.792157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.792128    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-data" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:31.792157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.792142    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:31.792354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.792214    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istiod-ca-cert" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:31.794203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.794149    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-kube-api-access-fm2fs" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "kube-api-access-fm2fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:31.794320 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.794206    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-podinfo" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:24:31.794383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.794365    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-envoy" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:31.794428 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.794369    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-token" pod "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" (UID: "e83b2a65-5c23-4a43-aa87-ba0806f7b7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:31.893446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.893415    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:31.893446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.893440    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:31.893446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.893449    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:31.893654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.893459    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm2fs\" (UniqueName: \"kubernetes.io/projected/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-kube-api-access-fm2fs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:31.893654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.893469    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:31.893654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.893477    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:31.893654 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:31.893487    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:32.039703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.039670    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:32.158426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.158318    5250 generic.go:356] "Generic (PLEG): container finished" podID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerID="f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928" exitCode=0
Oct 30 06:24:32.158426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.158351    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"
Oct 30 06:24:32.158691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.158481    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" event={"ID":"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8","Type":"ContainerDied","Data":"f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928"}
Oct 30 06:24:32.158691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.158521    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td" event={"ID":"e83b2a65-5c23-4a43-aa87-ba0806f7b7a8","Type":"ContainerDied","Data":"8c654d9e7acaa85e11653d804370b907be26727b09cd15abe7dce416706213d3"}
Oct 30 06:24:32.158691 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.158547    5250 scope.go:122] "RemoveContainer" containerID="f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928"
Oct 30 06:24:32.170023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.169996    5250 scope.go:122] "RemoveContainer" containerID="f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928"
Oct 30 06:24:32.170343 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:32.170321    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928\": container with ID starting with f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928 not found: ID does not exist" containerID="f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928"
Oct 30 06:24:32.170414 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.170349    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928"} err="failed to get container status \"f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928\": rpc error: code = NotFound desc = could not find container \"f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928\": container with ID starting with f606841ee228bd2f0fc62a87678f113fade6c4b9ea93aebba242bbdf5b1f0928 not found: ID does not exist"
Oct 30 06:24:32.201967 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.201937    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:32.235832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.235799    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"]
Oct 30 06:24:32.249436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:32.244875    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-649td"]
Oct 30 06:24:32.373615 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:32.373577    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:24:33.278792 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:33.278755    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" path="/var/lib/kubelet/pods/e83b2a65-5c23-4a43-aa87-ba0806f7b7a8/volumes"
Oct 30 06:24:33.649192 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:33.649115    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main" containerID="cri-o://4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb" gracePeriod=45
Oct 30 06:24:33.658666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:33.658616    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" containerID="cri-o://c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9" gracePeriod=45
Oct 30 06:24:33.662786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:33.662760    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:33.916708 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:33.916672    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 06:24:33.919714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:33.919694    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 06:24:34.013543 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013506    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-dshm\") pod \"04314bc8-7661-4875-b931-94dcfdef9ee6\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") "
Oct 30 06:24:34.013823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013584    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e5f085-5e01-47a9-a1a8-322c88830138-tls-certs\" (UniqueName: \"kubernetes.io/secret/39e5f085-5e01-47a9-a1a8-322c88830138-tls-certs\") pod \"39e5f085-5e01-47a9-a1a8-322c88830138\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") "
Oct 30 06:24:34.013823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013634    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-model-cache\") pod \"04314bc8-7661-4875-b931-94dcfdef9ee6\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") "
Oct 30 06:24:34.013823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013662    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-home\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-home\") pod \"39e5f085-5e01-47a9-a1a8-322c88830138\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") "
Oct 30 06:24:34.013823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013707    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04314bc8-7661-4875-b931-94dcfdef9ee6-tls-certs\" (UniqueName: \"kubernetes.io/secret/04314bc8-7661-4875-b931-94dcfdef9ee6-tls-certs\") pod \"04314bc8-7661-4875-b931-94dcfdef9ee6\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") "
Oct 30 06:24:34.013823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013731    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-home\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-home\") pod \"04314bc8-7661-4875-b931-94dcfdef9ee6\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") "
Oct 30 06:24:34.013823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013794    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-tmp-dir\") pod \"04314bc8-7661-4875-b931-94dcfdef9ee6\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") "
Oct 30 06:24:34.014403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013849    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e5f085-5e01-47a9-a1a8-322c88830138-kube-api-access-bnww8\" (UniqueName: \"kubernetes.io/projected/39e5f085-5e01-47a9-a1a8-322c88830138-kube-api-access-bnww8\") pod \"39e5f085-5e01-47a9-a1a8-322c88830138\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") "
Oct 30 06:24:34.014403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013916    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-dshm\") pod \"39e5f085-5e01-47a9-a1a8-322c88830138\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") "
Oct 30 06:24:34.014403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013947    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-tmp-dir\") pod \"39e5f085-5e01-47a9-a1a8-322c88830138\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") "
Oct 30 06:24:34.014403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013988    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-model-cache\") pod \"39e5f085-5e01-47a9-a1a8-322c88830138\" (UID: \"39e5f085-5e01-47a9-a1a8-322c88830138\") "
Oct 30 06:24:34.014403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.014017    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04314bc8-7661-4875-b931-94dcfdef9ee6-kube-api-access-8hh6f\" (UniqueName: \"kubernetes.io/projected/04314bc8-7661-4875-b931-94dcfdef9ee6-kube-api-access-8hh6f\") pod \"04314bc8-7661-4875-b931-94dcfdef9ee6\" (UID: \"04314bc8-7661-4875-b931-94dcfdef9ee6\") "
Oct 30 06:24:34.016025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.013909    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-model-cache" pod "04314bc8-7661-4875-b931-94dcfdef9ee6" (UID: "04314bc8-7661-4875-b931-94dcfdef9ee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:34.016025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.014135    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-home" pod "04314bc8-7661-4875-b931-94dcfdef9ee6" (UID: "04314bc8-7661-4875-b931-94dcfdef9ee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:34.016025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.015275    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-tmp-dir" pod "39e5f085-5e01-47a9-a1a8-322c88830138" (UID: "39e5f085-5e01-47a9-a1a8-322c88830138"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:34.016025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.015536    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-model-cache" pod "39e5f085-5e01-47a9-a1a8-322c88830138" (UID: "39e5f085-5e01-47a9-a1a8-322c88830138"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:34.016488 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.016063    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-tmp-dir" pod "04314bc8-7661-4875-b931-94dcfdef9ee6" (UID: "04314bc8-7661-4875-b931-94dcfdef9ee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:34.016488 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.016136    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-home" pod "39e5f085-5e01-47a9-a1a8-322c88830138" (UID: "39e5f085-5e01-47a9-a1a8-322c88830138"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:34.026803 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.026562    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e5f085-5e01-47a9-a1a8-322c88830138-kube-api-access-bnww8" pod "39e5f085-5e01-47a9-a1a8-322c88830138" (UID: "39e5f085-5e01-47a9-a1a8-322c88830138"). InnerVolumeSpecName "kube-api-access-bnww8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:34.026803 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.026686    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-dshm" pod "39e5f085-5e01-47a9-a1a8-322c88830138" (UID: "39e5f085-5e01-47a9-a1a8-322c88830138"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:34.026803 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.026774    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04314bc8-7661-4875-b931-94dcfdef9ee6-kube-api-access-8hh6f" pod "04314bc8-7661-4875-b931-94dcfdef9ee6" (UID: "04314bc8-7661-4875-b931-94dcfdef9ee6"). InnerVolumeSpecName "kube-api-access-8hh6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:34.027046 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.026945    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e5f085-5e01-47a9-a1a8-322c88830138-tls-certs" pod "39e5f085-5e01-47a9-a1a8-322c88830138" (UID: "39e5f085-5e01-47a9-a1a8-322c88830138"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:34.027846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.027818    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04314bc8-7661-4875-b931-94dcfdef9ee6-tls-certs" pod "04314bc8-7661-4875-b931-94dcfdef9ee6" (UID: "04314bc8-7661-4875-b931-94dcfdef9ee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:24:34.029378 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.029357    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-dshm" pod "04314bc8-7661-4875-b931-94dcfdef9ee6" (UID: "04314bc8-7661-4875-b931-94dcfdef9ee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:34.115058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115024    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115052    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115061    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04314bc8-7661-4875-b931-94dcfdef9ee6-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115068    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115075    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115099    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnww8\" (UniqueName: \"kubernetes.io/projected/39e5f085-5e01-47a9-a1a8-322c88830138-kube-api-access-bnww8\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115112    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115120    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115128    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e5f085-5e01-47a9-a1a8-322c88830138-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115138    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hh6f\" (UniqueName: \"kubernetes.io/projected/04314bc8-7661-4875-b931-94dcfdef9ee6-kube-api-access-8hh6f\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115145    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04314bc8-7661-4875-b931-94dcfdef9ee6-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.115383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.115154    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e5f085-5e01-47a9-a1a8-322c88830138-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:34.172169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.172082    5250 generic.go:356] "Generic (PLEG): container finished" podID="39e5f085-5e01-47a9-a1a8-322c88830138" containerID="c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9" exitCode=0
Oct 30 06:24:34.172169 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.172142    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"
Oct 30 06:24:34.172439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.172197    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" event={"ID":"39e5f085-5e01-47a9-a1a8-322c88830138","Type":"ContainerDied","Data":"c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9"}
Oct 30 06:24:34.172439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.172243    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" event={"ID":"39e5f085-5e01-47a9-a1a8-322c88830138","Type":"ContainerDied","Data":"671e508cd207bf29a04f5b2863fe43f6cb37b954ff38e5ca9d4dc8cf171e9b8f"}
Oct 30 06:24:34.172439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.172268    5250 scope.go:122] "RemoveContainer" containerID="c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9"
Oct 30 06:24:34.173619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.173599    5250 generic.go:356] "Generic (PLEG): container finished" podID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerID="4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb" exitCode=0
Oct 30 06:24:34.173743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.173638    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"
Oct 30 06:24:34.173743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.173653    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" event={"ID":"04314bc8-7661-4875-b931-94dcfdef9ee6","Type":"ContainerDied","Data":"4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb"}
Oct 30 06:24:34.173743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.173684    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb" event={"ID":"04314bc8-7661-4875-b931-94dcfdef9ee6","Type":"ContainerDied","Data":"cd16f28f8d307c9ede22134108498f8990f8d23b5c32232636662c847de985a3"}
Oct 30 06:24:34.181935 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.181915    5250 scope.go:122] "RemoveContainer" containerID="c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9"
Oct 30 06:24:34.182218 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:34.182201    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9\": container with ID starting with c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9 not found: ID does not exist" containerID="c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9"
Oct 30 06:24:34.182296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.182225    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9"} err="failed to get container status \"c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9\": rpc error: code = NotFound desc = could not find container \"c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9\": container with ID starting with c05e9a00dce07dbdc0fa2e932aa1c0c67be7e7d4bc09629f03b62cd995a324e9 not found: ID does not exist"
Oct 30 06:24:34.182296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.182240    5250 scope.go:122] "RemoveContainer" containerID="4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb"
Oct 30 06:24:34.191026 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.191009    5250 scope.go:122] "RemoveContainer" containerID="4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb"
Oct 30 06:24:34.191332 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:34.191312    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb\": container with ID starting with 4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb not found: ID does not exist" containerID="4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb"
Oct 30 06:24:34.191391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.191340    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb"} err="failed to get container status \"4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb\": rpc error: code = NotFound desc = could not find container \"4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb\": container with ID starting with 4c4d97df8242fa0f99a572f9e8f5fdcbc952c70f0c75c2ac1fe6ca2dea3956cb not found: ID does not exist"
Oct 30 06:24:34.217846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.217822    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"]
Oct 30 06:24:34.220619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.220597    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964"]
Oct 30 06:24:34.249888 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.249855    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"]
Oct 30 06:24:34.257385 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.257347    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/premium-w5-be78f2-kserve-599bc94ff9-898tb"]
Oct 30 06:24:34.657959 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.657923    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8" podUID="f40cfd76-bd78-423e-8ad7-ef3fee34eb52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:24:34.887917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:34.887860    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/simulator-w5-be78f2-kserve-7bdc44476b-82964" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 06:24:35.279159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:35.279125    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" path="/var/lib/kubelet/pods/04314bc8-7661-4875-b931-94dcfdef9ee6/volumes"
Oct 30 06:24:35.279569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:35.279555    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" path="/var/lib/kubelet/pods/39e5f085-5e01-47a9-a1a8-322c88830138/volumes"
Oct 30 06:24:36.489931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:36.489896    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:36.629714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:36.629676    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:36.782762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:36.782660    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"]
Oct 30 06:24:36.782943 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:36.782898    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy" containerID="cri-o://80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8" gracePeriod=30
Oct 30 06:24:36.865212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:36.865148    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:37.609582 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:37.609548    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:37.762767 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:37.762730    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:38.261725 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:38.261696    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:38.371645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:38.371608    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"]
Oct 30 06:24:38.380552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:38.380517    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-be78f2-wt6qw"]
Oct 30 06:24:38.408649 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:38.408619    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:38.888598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:38.888563    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:38.905659 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:38.905619    5250 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 06:24:38.905816 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:38.905690    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 06:24:39.279541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:39.279461    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3f2fb6-38fc-4601-af66-eac4ce2c4c30" path="/var/lib/kubelet/pods/da3f2fb6-38fc-4601-af66-eac4ce2c4c30/volumes"
Oct 30 06:24:39.502137 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:39.502101    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:42.040567 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.040538    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 06:24:42.192755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.192716    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-credential-socket\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.192755 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.192754    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-envoy\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.193021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.192787    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-socket\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.193021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.192811    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-istio-token\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-istio-token\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.193021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.192855    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-certs\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.193021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.192883    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-data\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.193021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.192923    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-kube-api-access-dzg56\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-kube-api-access-dzg56\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.193021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.192943    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ede062-4c03-4bf5-b803-01ce16478c33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ede062-4c03-4bf5-b803-01ce16478c33-istiod-ca-cert\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.193021 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.193007    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91ede062-4c03-4bf5-b803-01ce16478c33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ede062-4c03-4bf5-b803-01ce16478c33-istio-podinfo\") pod \"91ede062-4c03-4bf5-b803-01ce16478c33\" (UID: \"91ede062-4c03-4bf5-b803-01ce16478c33\") "
Oct 30 06:24:42.193439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.193043    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-credential-socket" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:42.193439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.193313    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:42.193439 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.193425    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-data" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:42.193613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.193453    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ede062-4c03-4bf5-b803-01ce16478c33-istiod-ca-cert" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:42.193613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.193480    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-certs" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:42.193613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.193525    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-socket" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:42.195232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.195152    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-envoy" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:42.195599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.195570    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-istio-token" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:42.195866 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.195841    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ede062-4c03-4bf5-b803-01ce16478c33-istio-podinfo" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:24:42.195940 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.195877    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-kube-api-access-dzg56" pod "91ede062-4c03-4bf5-b803-01ce16478c33" (UID: "91ede062-4c03-4bf5-b803-01ce16478c33"). InnerVolumeSpecName "kube-api-access-dzg56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:42.224411 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.224384    5250 generic.go:356] "Generic (PLEG): container finished" podID="91ede062-4c03-4bf5-b803-01ce16478c33" containerID="80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8" exitCode=0
Oct 30 06:24:42.224544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.224424    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" event={"ID":"91ede062-4c03-4bf5-b803-01ce16478c33","Type":"ContainerDied","Data":"80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8"}
Oct 30 06:24:42.224544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.224436    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"
Oct 30 06:24:42.224544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.224467    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq" event={"ID":"91ede062-4c03-4bf5-b803-01ce16478c33","Type":"ContainerDied","Data":"8e10f1e92f8bc5db7216abd610704b7ff4a509db3975aea4068e21da8ca14e60"}
Oct 30 06:24:42.224544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.224489    5250 scope.go:122] "RemoveContainer" containerID="80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8"
Oct 30 06:24:42.236082 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.236059    5250 scope.go:122] "RemoveContainer" containerID="80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8"
Oct 30 06:24:42.236442 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:42.236422    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8\": container with ID starting with 80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8 not found: ID does not exist" containerID="80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8"
Oct 30 06:24:42.236535 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.236451    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8"} err="failed to get container status \"80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8\": rpc error: code = NotFound desc = could not find container \"80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8\": container with ID starting with 80a013022db7460648faffc9a5be7a9071debf3ceaa635727eb24505e76abee8 not found: ID does not exist"
Oct 30 06:24:42.269417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.269385    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"]
Oct 30 06:24:42.275916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.275891    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-vpjcq"]
Oct 30 06:24:42.293869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.293835    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzg56\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-kube-api-access-dzg56\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:42.293977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.293869    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ede062-4c03-4bf5-b803-01ce16478c33-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:42.293977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.293889    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ede062-4c03-4bf5-b803-01ce16478c33-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:42.293977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.293905    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:42.293977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.293920    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:42.293977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.293934    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ede062-4c03-4bf5-b803-01ce16478c33-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:42.293977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.293948    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:42.293977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:42.293961    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ede062-4c03-4bf5-b803-01ce16478c33-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:43.278928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:43.278895    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" path="/var/lib/kubelet/pods/91ede062-4c03-4bf5-b803-01ce16478c33/volumes"
Oct 30 06:24:43.933961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:43.933924    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:44.081420 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:44.081392    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:44.657822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:44.657787    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8" podUID="f40cfd76-bd78-423e-8ad7-ef3fee34eb52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 06:24:45.945042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:45.945008    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:46.087719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:46.087682    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:46.289220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:46.289099    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"]
Oct 30 06:24:46.289463 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:46.289430    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy" containerID="cri-o://0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e" gracePeriod=30
Oct 30 06:24:46.352825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:46.352783    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:49.461326 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:49.461301    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 06:24:49.557632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:49.557557    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-plugins-config-volume\") pod \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\" (UID: \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\") "
Oct 30 06:24:49.557632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:49.557612    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-kube-api-access-s5rwg\" (UniqueName: \"kubernetes.io/projected/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-kube-api-access-s5rwg\") pod \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\" (UID: \"f40cfd76-bd78-423e-8ad7-ef3fee34eb52\") "
Oct 30 06:24:49.557988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:49.557965    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-plugins-config-volume" pod "f40cfd76-bd78-423e-8ad7-ef3fee34eb52" (UID: "f40cfd76-bd78-423e-8ad7-ef3fee34eb52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:49.559858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:49.559839    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-kube-api-access-s5rwg" pod "f40cfd76-bd78-423e-8ad7-ef3fee34eb52" (UID: "f40cfd76-bd78-423e-8ad7-ef3fee34eb52"). InnerVolumeSpecName "kube-api-access-s5rwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:49.658568 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:49.658538    5250 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-plugins-config-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:49.658568 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:49.658565    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5rwg\" (UniqueName: \"kubernetes.io/projected/f40cfd76-bd78-423e-8ad7-ef3fee34eb52-kube-api-access-s5rwg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:50.271542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.271511    5250 generic.go:356] "Generic (PLEG): container finished" podID="f40cfd76-bd78-423e-8ad7-ef3fee34eb52" containerID="6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2" exitCode=0
Oct 30 06:24:50.271705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.271559    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"
Oct 30 06:24:50.271705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.271591    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8" event={"ID":"f40cfd76-bd78-423e-8ad7-ef3fee34eb52","Type":"ContainerDied","Data":"6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2"}
Oct 30 06:24:50.271705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.271647    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8" event={"ID":"f40cfd76-bd78-423e-8ad7-ef3fee34eb52","Type":"ContainerDied","Data":"5a45e3b3e1b9c33e7f1145ad728f4870c9c3874e4640591b10e84ba0fab48293"}
Oct 30 06:24:50.271705 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.271671    5250 scope.go:122] "RemoveContainer" containerID="6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2"
Oct 30 06:24:50.281353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.281326    5250 scope.go:122] "RemoveContainer" containerID="6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2"
Oct 30 06:24:50.281602 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:50.281584    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2\": container with ID starting with 6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2 not found: ID does not exist" containerID="6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2"
Oct 30 06:24:50.281643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.281612    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2"} err="failed to get container status \"6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2\": rpc error: code = NotFound desc = could not find container \"6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2\": container with ID starting with 6a6beac043b6d287cf4868c3ace290e2984f74ffad2474659ad255a41dcacbf2 not found: ID does not exist"
Oct 30 06:24:50.316397 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.316377    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"]
Oct 30 06:24:50.326244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:50.326222    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-be78f2-64967b8fc9-r5kz8"]
Oct 30 06:24:51.280521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.280484    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40cfd76-bd78-423e-8ad7-ef3fee34eb52" path="/var/lib/kubelet/pods/f40cfd76-bd78-423e-8ad7-ef3fee34eb52/volumes"
Oct 30 06:24:51.557805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.557773    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 06:24:51.678840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.678787    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-data\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.678848    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-certs\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.678904    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-socket\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.678965    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-kube-api-access-2kgbp\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-kube-api-access-2kgbp\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679004    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-podinfo\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679078    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-envoy\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679109    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-certs" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:51.679359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679125    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-credential-socket\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679139    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-data" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:51.679359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679197    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b436d1e3-a390-48b6-8a1c-493850bf5817-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b436d1e3-a390-48b6-8a1c-493850bf5817-istiod-ca-cert\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679228    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-token\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-token\") pod \"b436d1e3-a390-48b6-8a1c-493850bf5817\" (UID: \"b436d1e3-a390-48b6-8a1c-493850bf5817\") "
Oct 30 06:24:51.679729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679516    5250 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-data\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.679729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679536    5250 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.679729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679616    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-credential-socket" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:51.679994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679961    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-socket" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:51.680109 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.679993    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b436d1e3-a390-48b6-8a1c-493850bf5817-istiod-ca-cert" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:24:51.682007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.681981    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-podinfo" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 06:24:51.682265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.682242    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-token" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:51.682520 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.682487    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-kube-api-access-2kgbp" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "kube-api-access-2kgbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:24:51.683215 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.683164    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-envoy" pod "b436d1e3-a390-48b6-8a1c-493850bf5817" (UID: "b436d1e3-a390-48b6-8a1c-493850bf5817"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:24:51.780785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.780741    5250 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-credential-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.780785 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.780780    5250 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b436d1e3-a390-48b6-8a1c-493850bf5817-istiod-ca-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.781004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.780796    5250 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.781004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.780809    5250 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-workload-socket\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.781004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.780824    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kgbp\" (UniqueName: \"kubernetes.io/projected/b436d1e3-a390-48b6-8a1c-493850bf5817-kube-api-access-2kgbp\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.781004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.780838    5250 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-podinfo\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.781004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.780851    5250 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b436d1e3-a390-48b6-8a1c-493850bf5817-istio-envoy\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:24:51.869645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:51.869547    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:52.288681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.288640    5250 generic.go:356] "Generic (PLEG): container finished" podID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerID="0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e" exitCode=0
Oct 30 06:24:52.289246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.288728    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"
Oct 30 06:24:52.289246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.288736    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" event={"ID":"b436d1e3-a390-48b6-8a1c-493850bf5817","Type":"ContainerDied","Data":"0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e"}
Oct 30 06:24:52.289246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.288812    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns" event={"ID":"b436d1e3-a390-48b6-8a1c-493850bf5817","Type":"ContainerDied","Data":"7a4fb5e3cd4f30b0fc9e216c03993ed76c316c27cc9a31346f874127ca32a170"}
Oct 30 06:24:52.289246 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.288847    5250 scope.go:122] "RemoveContainer" containerID="0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e"
Oct 30 06:24:52.301663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.301640    5250 scope.go:122] "RemoveContainer" containerID="0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e"
Oct 30 06:24:52.301992 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:24:52.301964    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e\": container with ID starting with 0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e not found: ID does not exist" containerID="0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e"
Oct 30 06:24:52.302101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.302004    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e"} err="failed to get container status \"0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e\": rpc error: code = NotFound desc = could not find container \"0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e\": container with ID starting with 0cfd3a01980da8e486ea20f6937bb9f3f5d2b0da7a707220fea9d7a88d1c534e not found: ID does not exist"
Oct 30 06:24:52.344552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.344524    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"]
Oct 30 06:24:52.354287 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:52.354265    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-89lns"]
Oct 30 06:24:53.279684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:53.279651    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" path="/var/lib/kubelet/pods/b436d1e3-a390-48b6-8a1c-493850bf5817/volumes"
Oct 30 06:24:54.947556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:54.947520    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:55.091778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:55.091748    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:56.883932 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:56.883891    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:24:57.028931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:57.028898    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"]
Oct 30 06:24:57.038688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:57.038661    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40e345-hbcq2"]
Oct 30 06:24:57.279008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:24:57.278918    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7617e8-53a7-4240-9cf3-03aceedc9153" path="/var/lib/kubelet/pods/ea7617e8-53a7-4240-9cf3-03aceedc9153/volumes"
Oct 30 06:25:02.373396 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:25:02.373351    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:25:02.559617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:02.559579    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:25:14.392606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.392558    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"]
Oct 30 06:25:14.393112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393052    5250 cpu_manager.go:414] "containerMap: removing container" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main"
Oct 30 06:25:14.393112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393071    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main"
Oct 30 06:25:14.393112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393098    5250 cpu_manager.go:414] "containerMap: removing container" podUID="f40cfd76-bd78-423e-8ad7-ef3fee34eb52" containerName="payload-pre-processing"
Oct 30 06:25:14.393112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393106    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40cfd76-bd78-423e-8ad7-ef3fee34eb52" containerName="payload-pre-processing"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393117    5250 cpu_manager.go:414] "containerMap: removing container" podUID="9ce112f1-37d7-4061-bc37-dcab95805834" containerName="payload-processing"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393124    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce112f1-37d7-4061-bc37-dcab95805834" containerName="payload-processing"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393138    5250 cpu_manager.go:414] "containerMap: removing container" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393146    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393163    5250 cpu_manager.go:414] "containerMap: removing container" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393172    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393205    5250 cpu_manager.go:414] "containerMap: removing container" podUID="311098d1-3cd9-476d-9f9c-3b57a69da5a6" containerName="revoke-keys"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393213    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311098d1-3cd9-476d-9f9c-3b57a69da5a6" containerName="revoke-keys"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393226    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393233    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393260    5250 cpu_manager.go:414] "containerMap: removing container" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393267    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393352    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40cfd76-bd78-423e-8ad7-ef3fee34eb52" containerName="payload-pre-processing"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393366    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce112f1-37d7-4061-bc37-dcab95805834" containerName="payload-processing"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393379    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ede062-4c03-4bf5-b803-01ce16478c33" containerName="istio-proxy"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393390    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="04314bc8-7661-4875-b931-94dcfdef9ee6" containerName="main"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393401    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="311098d1-3cd9-476d-9f9c-3b57a69da5a6" containerName="revoke-keys"
Oct 30 06:25:14.393402 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393409    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83b2a65-5c23-4a43-aa87-ba0806f7b7a8" containerName="istio-proxy"
Oct 30 06:25:14.394388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393421    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="b436d1e3-a390-48b6-8a1c-493850bf5817" containerName="istio-proxy"
Oct 30 06:25:14.394388 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.393431    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e5f085-5e01-47a9-a1a8-322c88830138" containerName="main"
Oct 30 06:25:14.396906 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.396887    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.416110 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.416087    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:25:14.416240 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.416122    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:25:14.428159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.428136    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"]
Oct 30 06:25:14.462303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.462268    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.462460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.462402    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.462460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.462436    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn45t\" (UniqueName: \"kubernetes.io/projected/61d2541c-0924-4952-b756-eabe1c81cd83-kube-api-access-wn45t\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.563577 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.563546    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn45t\" (UniqueName: \"kubernetes.io/projected/61d2541c-0924-4952-b756-eabe1c81cd83-kube-api-access-wn45t\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.563690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.563591    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.563790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.563766    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.564368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.564350    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.566278 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.566262    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.584012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.583994    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn45t\" (UniqueName: \"kubernetes.io/projected/61d2541c-0924-4952-b756-eabe1c81cd83-kube-api-access-wn45t\") pod \"maas-api-revoke-sub-85237e45-2x58z\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:14.709415 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:14.709353    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:15.007280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:15.007132    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"]
Oct 30 06:25:15.011675 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:25:15.011645    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d2541c_0924_4952_b756_eabe1c81cd83.slice/crio-d173bb3ad0b8e8e3b9e87c13a4bf94e9ed61a24d4774b48e1deaabbc7348a605 WatchSource:0}: Error finding container d173bb3ad0b8e8e3b9e87c13a4bf94e9ed61a24d4774b48e1deaabbc7348a605: Status 404 returned error can't find the container with id d173bb3ad0b8e8e3b9e87c13a4bf94e9ed61a24d4774b48e1deaabbc7348a605
Oct 30 06:25:15.416992 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:15.416957    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z" event={"ID":"61d2541c-0924-4952-b756-eabe1c81cd83","Type":"ContainerStarted","Data":"9d83ac37dab0c139271123ee98bd826b10460827bdd1803a794bb301e49f8c5d"}
Oct 30 06:25:15.417368 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:15.417020    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z" event={"ID":"61d2541c-0924-4952-b756-eabe1c81cd83","Type":"ContainerStarted","Data":"d173bb3ad0b8e8e3b9e87c13a4bf94e9ed61a24d4774b48e1deaabbc7348a605"}
Oct 30 06:25:15.453477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:15.453422    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z" podStartSLOduration=1.453404455 podStartE2EDuration="1.453404455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:15.451582614 +0000 UTC m=+3230.282619830" watchObservedRunningTime="2026-10-30 06:25:15.453404455 +0000 UTC m=+3230.284441673"
Oct 30 06:25:16.424156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:16.424115    5250 generic.go:356] "Generic (PLEG): container finished" podID="61d2541c-0924-4952-b756-eabe1c81cd83" containerID="9d83ac37dab0c139271123ee98bd826b10460827bdd1803a794bb301e49f8c5d" exitCode=0
Oct 30 06:25:16.424665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:16.424214    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z" event={"ID":"61d2541c-0924-4952-b756-eabe1c81cd83","Type":"ContainerDied","Data":"9d83ac37dab0c139271123ee98bd826b10460827bdd1803a794bb301e49f8c5d"}
Oct 30 06:25:17.558548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.558526    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:17.694625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.694545    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-xks-ca\") pod \"61d2541c-0924-4952-b756-eabe1c81cd83\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") "
Oct 30 06:25:17.694789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.694651    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-openshift-ca\") pod \"61d2541c-0924-4952-b756-eabe1c81cd83\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") "
Oct 30 06:25:17.694789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.694677    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d2541c-0924-4952-b756-eabe1c81cd83-kube-api-access-wn45t\" (UniqueName: \"kubernetes.io/projected/61d2541c-0924-4952-b756-eabe1c81cd83-kube-api-access-wn45t\") pod \"61d2541c-0924-4952-b756-eabe1c81cd83\" (UID: \"61d2541c-0924-4952-b756-eabe1c81cd83\") "
Oct 30 06:25:17.695035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.695005    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-openshift-ca" pod "61d2541c-0924-4952-b756-eabe1c81cd83" (UID: "61d2541c-0924-4952-b756-eabe1c81cd83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:25:17.696994 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.696965    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d2541c-0924-4952-b756-eabe1c81cd83-kube-api-access-wn45t" pod "61d2541c-0924-4952-b756-eabe1c81cd83" (UID: "61d2541c-0924-4952-b756-eabe1c81cd83"). InnerVolumeSpecName "kube-api-access-wn45t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:25:17.697120 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.697016    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-xks-ca" pod "61d2541c-0924-4952-b756-eabe1c81cd83" (UID: "61d2541c-0924-4952-b756-eabe1c81cd83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:25:17.796104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.796077    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:25:17.796104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.796101    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn45t\" (UniqueName: \"kubernetes.io/projected/61d2541c-0924-4952-b756-eabe1c81cd83-kube-api-access-wn45t\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:25:17.796104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:17.796110    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2541c-0924-4952-b756-eabe1c81cd83-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:25:18.436900 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:18.436865    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z" event={"ID":"61d2541c-0924-4952-b756-eabe1c81cd83","Type":"ContainerDied","Data":"d173bb3ad0b8e8e3b9e87c13a4bf94e9ed61a24d4774b48e1deaabbc7348a605"}
Oct 30 06:25:18.437098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:18.436913    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d173bb3ad0b8e8e3b9e87c13a4bf94e9ed61a24d4774b48e1deaabbc7348a605"
Oct 30 06:25:18.437098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:18.436874    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"
Oct 30 06:25:25.435061 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.435028    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:25:25.587713 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.587678    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"]
Oct 30 06:25:25.588114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.588097    5250 cpu_manager.go:414] "containerMap: removing container" podUID="61d2541c-0924-4952-b756-eabe1c81cd83" containerName="revoke-keys"
Oct 30 06:25:25.588166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.588117    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d2541c-0924-4952-b756-eabe1c81cd83" containerName="revoke-keys"
Oct 30 06:25:25.588234 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.588206    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d2541c-0924-4952-b756-eabe1c81cd83" containerName="revoke-keys"
Oct 30 06:25:25.591462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.591447    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.616356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.616312    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"]
Oct 30 06:25:25.661917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.661871    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64k8\" (UniqueName: \"kubernetes.io/projected/19cd4d21-143f-43b5-b230-0b82140a1567-kube-api-access-c64k8\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.661917 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.661912    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cd4d21-143f-43b5-b230-0b82140a1567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.662166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.662034    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cd4d21-143f-43b5-b230-0b82140a1567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.662166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.662066    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cd4d21-143f-43b5-b230-0b82140a1567-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.662166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.662098    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.662166 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.662132    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.662375 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.662253    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.662375 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.662278    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.662375 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.662299    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.763795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.763717    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64k8\" (UniqueName: \"kubernetes.io/projected/19cd4d21-143f-43b5-b230-0b82140a1567-kube-api-access-c64k8\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.763795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.763754    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cd4d21-143f-43b5-b230-0b82140a1567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.763804    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cd4d21-143f-43b5-b230-0b82140a1567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.763827    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cd4d21-143f-43b5-b230-0b82140a1567-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.763952    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764012 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.763988    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.764058    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.764095    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764275 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.764135    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.764420    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.764441    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764552 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.764509    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764598 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.764577    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.764637 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.764608    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cd4d21-143f-43b5-b230-0b82140a1567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.766511 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.766494    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cd4d21-143f-43b5-b230-0b82140a1567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.766763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.766746    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cd4d21-143f-43b5-b230-0b82140a1567-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.784735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.784710    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cd4d21-143f-43b5-b230-0b82140a1567-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.784927 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.784909    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64k8\" (UniqueName: \"kubernetes.io/projected/19cd4d21-143f-43b5-b230-0b82140a1567-kube-api-access-c64k8\") pod \"maas-default-gateway-openshift-default-84587756dc-x56kg\" (UID: \"19cd4d21-143f-43b5-b230-0b82140a1567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.905148 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.905107    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:25.960686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:25.960375    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:25:26.222008 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:26.221972    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"]
Oct 30 06:25:26.230354 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:26.230320    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:25:26.230462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:26.230392    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:25:26.230462 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:26.230420    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:25:26.481861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:26.481772    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" event={"ID":"19cd4d21-143f-43b5-b230-0b82140a1567","Type":"ContainerStarted","Data":"c43bc0f835da4130d20f3d11d48c6ec7f37216f11692b63aea4389a080fc4116"}
Oct 30 06:25:26.481861 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:26.481822    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" event={"ID":"19cd4d21-143f-43b5-b230-0b82140a1567","Type":"ContainerStarted","Data":"bfee8cb78fa5ddbf852598b1e3f6c4993c490761b63cb6d90cff786401504aea"}
Oct 30 06:25:26.525645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:26.525597    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" podStartSLOduration=1.52558208 podStartE2EDuration="1.52558208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:26.523486797 +0000 UTC m=+3241.354524013" watchObservedRunningTime="2026-10-30 06:25:26.52558208 +0000 UTC m=+3241.356619298"
Oct 30 06:25:26.905480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:26.905445    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:27.819022 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:27.818985    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:25:27.906501 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:27.906468    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-x56kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 06:25:27.906704 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:27.906525    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" podUID="19cd4d21-143f-43b5-b230-0b82140a1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 06:25:28.906248 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:28.906211    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-x56kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 06:25:28.906662 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:28.906265    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" podUID="19cd4d21-143f-43b5-b230-0b82140a1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 06:25:29.590106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:29.590070    5250 scope.go:122] "RemoveContainer" containerID="b44274c6c27feced8964bc6c6eeefcd69526435a567173abb20aea764960f8f1"
Oct 30 06:25:29.599308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:29.599284    5250 scope.go:122] "RemoveContainer" containerID="12ff6f703d1aad834c468260bce13bfef0490d3149f8b4ec722962731ba6fb89"
Oct 30 06:25:29.608142 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:29.608119    5250 scope.go:122] "RemoveContainer" containerID="88a9d85195ff0246d20c83d01dac105f180f139ab0f63905255a4e2d0b1ebfc4"
Oct 30 06:25:29.906097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:29.906063    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-x56kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 06:25:29.906297 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:29.906118    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" podUID="19cd4d21-143f-43b5-b230-0b82140a1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 06:25:30.905955 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:30.905920    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-x56kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 06:25:30.906204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:30.905982    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" podUID="19cd4d21-143f-43b5-b230-0b82140a1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 06:25:31.905672 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:31.905635    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-x56kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 06:25:31.906059 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:31.905691    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" podUID="19cd4d21-143f-43b5-b230-0b82140a1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 06:25:32.372522 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:25:32.372436    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:25:32.905534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:32.905495    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-x56kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 06:25:32.905735 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:32.905553    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg" podUID="19cd4d21-143f-43b5-b230-0b82140a1567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 06:25:34.399266 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:34.399237    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:34.399793 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:34.399768    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:34.400445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:34.400424    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-x56kg"
Oct 30 06:25:37.718584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:37.718501    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:25:38.179281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:38.179251    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:25:44.754693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:44.754657    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:25:44.900442 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:25:44.900370    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:26:02.288965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.288928    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"]
Oct 30 06:26:02.292353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.292330    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.314683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.314653    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:26:02.314683 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.314659    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:26:02.316952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.316929    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"]
Oct 30 06:26:02.373123 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:26:02.373082    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:26:02.483040 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.483009    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlmg5\" (UniqueName: \"kubernetes.io/projected/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-kube-api-access-nlmg5\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.483233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.483060    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.483233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.483109    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.584185 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.584105    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmg5\" (UniqueName: \"kubernetes.io/projected/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-kube-api-access-nlmg5\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.584185 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.584149    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.584385 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.584173    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.584887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.584867    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.586730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.586713    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.605548 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.605523    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlmg5\" (UniqueName: \"kubernetes.io/projected/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-kube-api-access-nlmg5\") pod \"maas-api-revoke-sub-c1e271a9-cn2jx\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:02.905905 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:02.905869    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:03.202926 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:03.202842    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"]
Oct 30 06:26:03.206144 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:26:03.206114    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9e7b99_aad7_40e1_a8f5_04b2ccd70ee0.slice/crio-b4cc7b0af43f298b7f6a4c0f6c6e99ebdd8cd587d8c56a7ea46d910caa0e0e4c WatchSource:0}: Error finding container b4cc7b0af43f298b7f6a4c0f6c6e99ebdd8cd587d8c56a7ea46d910caa0e0e4c: Status 404 returned error can't find the container with id b4cc7b0af43f298b7f6a4c0f6c6e99ebdd8cd587d8c56a7ea46d910caa0e0e4c
Oct 30 06:26:03.708825 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:03.708782    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx" event={"ID":"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0","Type":"ContainerStarted","Data":"d082e3dadc8f012ab71edfb6d436b93d14e0b6d32553be60701778a11ad22144"}
Oct 30 06:26:03.709316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:03.708875    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx" event={"ID":"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0","Type":"ContainerStarted","Data":"b4cc7b0af43f298b7f6a4c0f6c6e99ebdd8cd587d8c56a7ea46d910caa0e0e4c"}
Oct 30 06:26:03.744992 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:03.744942    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx" podStartSLOduration=1.744926974 podStartE2EDuration="1.744926974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:03.744253111 +0000 UTC m=+3278.575290330" watchObservedRunningTime="2026-10-30 06:26:03.744926974 +0000 UTC m=+3278.575964194"
Oct 30 06:26:04.715627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:04.715593    5250 generic.go:356] "Generic (PLEG): container finished" podID="be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0" containerID="d082e3dadc8f012ab71edfb6d436b93d14e0b6d32553be60701778a11ad22144" exitCode=0
Oct 30 06:26:04.715627 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:04.715628    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx" event={"ID":"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0","Type":"ContainerDied","Data":"d082e3dadc8f012ab71edfb6d436b93d14e0b6d32553be60701778a11ad22144"}
Oct 30 06:26:05.850739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:05.850715    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:05.916611 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:05.916574    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-openshift-ca\") pod \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") "
Oct 30 06:26:05.917031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:05.917004    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-openshift-ca" pod "be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0" (UID: "be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:26:06.017559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.017479    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-kube-api-access-nlmg5\" (UniqueName: \"kubernetes.io/projected/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-kube-api-access-nlmg5\") pod \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") "
Oct 30 06:26:06.017559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.017531    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-xks-ca\") pod \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\" (UID: \"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0\") "
Oct 30 06:26:06.017745 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.017667    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:06.019734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.019706    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-xks-ca" pod "be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0" (UID: "be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:26:06.019862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.019737    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-kube-api-access-nlmg5" pod "be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0" (UID: "be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0"). InnerVolumeSpecName "kube-api-access-nlmg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:26:06.118330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.118300    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlmg5\" (UniqueName: \"kubernetes.io/projected/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-kube-api-access-nlmg5\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:06.118330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.118327    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:06.727295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.727237    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx" event={"ID":"be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0","Type":"ContainerDied","Data":"b4cc7b0af43f298b7f6a4c0f6c6e99ebdd8cd587d8c56a7ea46d910caa0e0e4c"}
Oct 30 06:26:06.727295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.727283    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"
Oct 30 06:26:06.727295 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:06.727299    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4cc7b0af43f298b7f6a4c0f6c6e99ebdd8cd587d8c56a7ea46d910caa0e0e4c"
Oct 30 06:26:22.170827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.170789    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"]
Oct 30 06:26:22.171370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.171345    5250 cpu_manager.go:414] "containerMap: removing container" podUID="be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0" containerName="revoke-keys"
Oct 30 06:26:22.171370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.171364    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0" containerName="revoke-keys"
Oct 30 06:26:22.171498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.171468    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0" containerName="revoke-keys"
Oct 30 06:26:22.174674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.174649    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.189828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.189805    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:26:22.192823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.192796    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"]
Oct 30 06:26:22.200030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.200004    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:26:22.258810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.258770    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5wz\" (UniqueName: \"kubernetes.io/projected/024889b7-83fa-44d1-9421-600d8eeddf54-kube-api-access-gl5wz\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.258810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.258814    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.259022 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.258893    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.359413 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.359382    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.359580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.359473    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5wz\" (UniqueName: \"kubernetes.io/projected/024889b7-83fa-44d1-9421-600d8eeddf54-kube-api-access-gl5wz\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.359580 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.359499    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.360080 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.360058    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.361929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.361912    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.381377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.381347    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5wz\" (UniqueName: \"kubernetes.io/projected/024889b7-83fa-44d1-9421-600d8eeddf54-kube-api-access-gl5wz\") pod \"maas-api-revoke-sub-2e1f5b5f-tczw4\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.487550 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.487478    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:22.784953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.784882    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"]
Oct 30 06:26:22.788335 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:26:22.788307    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024889b7_83fa_44d1_9421_600d8eeddf54.slice/crio-4be5112ed8c4a63d873baf4ac9dfd266af5ee28631852bbf7cb5003018eb55d1 WatchSource:0}: Error finding container 4be5112ed8c4a63d873baf4ac9dfd266af5ee28631852bbf7cb5003018eb55d1: Status 404 returned error can't find the container with id 4be5112ed8c4a63d873baf4ac9dfd266af5ee28631852bbf7cb5003018eb55d1
Oct 30 06:26:22.827890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:22.827866    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4" event={"ID":"024889b7-83fa-44d1-9421-600d8eeddf54","Type":"ContainerStarted","Data":"4be5112ed8c4a63d873baf4ac9dfd266af5ee28631852bbf7cb5003018eb55d1"}
Oct 30 06:26:23.834970 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:23.834876    5250 generic.go:356] "Generic (PLEG): container finished" podID="024889b7-83fa-44d1-9421-600d8eeddf54" containerID="2ff12653b75df57f3b1addf78c43550b008cb9ef002618d6865d3283238a5301" exitCode=0
Oct 30 06:26:23.835444 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:23.834958    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4" event={"ID":"024889b7-83fa-44d1-9421-600d8eeddf54","Type":"ContainerDied","Data":"2ff12653b75df57f3b1addf78c43550b008cb9ef002618d6865d3283238a5301"}
Oct 30 06:26:24.975149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:24.975125    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:25.084417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.084382    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-xks-ca\") pod \"024889b7-83fa-44d1-9421-600d8eeddf54\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") "
Oct 30 06:26:25.084599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.084428    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-openshift-ca\") pod \"024889b7-83fa-44d1-9421-600d8eeddf54\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") "
Oct 30 06:26:25.084599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.084472    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024889b7-83fa-44d1-9421-600d8eeddf54-kube-api-access-gl5wz\" (UniqueName: \"kubernetes.io/projected/024889b7-83fa-44d1-9421-600d8eeddf54-kube-api-access-gl5wz\") pod \"024889b7-83fa-44d1-9421-600d8eeddf54\" (UID: \"024889b7-83fa-44d1-9421-600d8eeddf54\") "
Oct 30 06:26:25.084884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.084858    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-openshift-ca" pod "024889b7-83fa-44d1-9421-600d8eeddf54" (UID: "024889b7-83fa-44d1-9421-600d8eeddf54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:26:25.086662 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.086638    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-xks-ca" pod "024889b7-83fa-44d1-9421-600d8eeddf54" (UID: "024889b7-83fa-44d1-9421-600d8eeddf54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:26:25.086762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.086709    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024889b7-83fa-44d1-9421-600d8eeddf54-kube-api-access-gl5wz" pod "024889b7-83fa-44d1-9421-600d8eeddf54" (UID: "024889b7-83fa-44d1-9421-600d8eeddf54"). InnerVolumeSpecName "kube-api-access-gl5wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:26:25.185857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.185821    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:25.185857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.185852    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024889b7-83fa-44d1-9421-600d8eeddf54-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:25.185857 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.185862    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl5wz\" (UniqueName: \"kubernetes.io/projected/024889b7-83fa-44d1-9421-600d8eeddf54-kube-api-access-gl5wz\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:25.750304 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.750267    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"]
Oct 30 06:26:25.750710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.750634    5250 cpu_manager.go:414] "containerMap: removing container" podUID="024889b7-83fa-44d1-9421-600d8eeddf54" containerName="revoke-keys"
Oct 30 06:26:25.750710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.750655    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024889b7-83fa-44d1-9421-600d8eeddf54" containerName="revoke-keys"
Oct 30 06:26:25.750892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.750766    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="024889b7-83fa-44d1-9421-600d8eeddf54" containerName="revoke-keys"
Oct 30 06:26:25.753877 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.753859    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.778965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.778939    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"]
Oct 30 06:26:25.848454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.848425    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"
Oct 30 06:26:25.848642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.848462    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4" event={"ID":"024889b7-83fa-44d1-9421-600d8eeddf54","Type":"ContainerDied","Data":"4be5112ed8c4a63d873baf4ac9dfd266af5ee28631852bbf7cb5003018eb55d1"}
Oct 30 06:26:25.848642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.848516    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4be5112ed8c4a63d873baf4ac9dfd266af5ee28631852bbf7cb5003018eb55d1"
Oct 30 06:26:25.891335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891304    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.891472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891339    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.891472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891362    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.891472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891433    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.891588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891497    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbdwb\" (UniqueName: \"kubernetes.io/projected/18c212e1-b3df-43b8-b066-9ffb699837ed-kube-api-access-jbdwb\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.891588 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891527    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c212e1-b3df-43b8-b066-9ffb699837ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.891665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891596    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.891665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891624    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.891665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.891655    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992061 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992032    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c212e1-b3df-43b8-b066-9ffb699837ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992093    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992118    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992165    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992260    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992291    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992323    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992365    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992396    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbdwb\" (UniqueName: \"kubernetes.io/projected/18c212e1-b3df-43b8-b066-9ffb699837ed-kube-api-access-jbdwb\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.992590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992548    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.993032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992604    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.993032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992876    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c212e1-b3df-43b8-b066-9ffb699837ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.993032 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.992986    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.993220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.993202    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.995165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.995141    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:25.995391 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:25.995369    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:26.015034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.014973    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c212e1-b3df-43b8-b066-9ffb699837ed-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:26.015321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.015298    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbdwb\" (UniqueName: \"kubernetes.io/projected/18c212e1-b3df-43b8-b066-9ffb699837ed-kube-api-access-jbdwb\") pod \"maas-default-gateway-openshift-default-84587756dc-vp9dv\" (UID: \"18c212e1-b3df-43b8-b066-9ffb699837ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:26.016818 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.016802    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:26:26.066601 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.066578    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:26.222669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.222645    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:26:26.383387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.383274    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"]
Oct 30 06:26:26.386080 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:26:26.386053    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c212e1_b3df_43b8_b066_9ffb699837ed.slice/crio-ea9643eaa4ab109d34e99cfe81ed47aac97e8b1805c576f0865696ce16d504bf WatchSource:0}: Error finding container ea9643eaa4ab109d34e99cfe81ed47aac97e8b1805c576f0865696ce16d504bf: Status 404 returned error can't find the container with id ea9643eaa4ab109d34e99cfe81ed47aac97e8b1805c576f0865696ce16d504bf
Oct 30 06:26:26.388237 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.388207    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:26:26.388315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.388270    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:26:26.388315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.388299    5250 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 06:26:26.880436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.877294    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" event={"ID":"18c212e1-b3df-43b8-b066-9ffb699837ed","Type":"ContainerStarted","Data":"0960ff8c0eb80a65b1e6d458d69e21d6f6bc75556b3f2c1ef122604c76a22804"}
Oct 30 06:26:26.880436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.877368    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" event={"ID":"18c212e1-b3df-43b8-b066-9ffb699837ed","Type":"ContainerStarted","Data":"ea9643eaa4ab109d34e99cfe81ed47aac97e8b1805c576f0865696ce16d504bf"}
Oct 30 06:26:26.921064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:26.921006    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podStartSLOduration=1.9209920120000001 podStartE2EDuration="1.920992012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:26.918478745 +0000 UTC m=+3301.749515963" watchObservedRunningTime="2026-10-30 06:26:26.920992012 +0000 UTC m=+3301.752029229"
Oct 30 06:26:27.076204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:27.074601    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:27.080356 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:27.080315    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 06:26:27.080541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:27.080388    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 06:26:28.072632 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:28.072592    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 06:26:28.072919 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:28.072890    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 06:26:29.076206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:29.070597    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 06:26:29.076206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:29.070657    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 06:26:29.685408 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:29.685379    5250 scope.go:122] "RemoveContainer" containerID="99ad8722db3598147a68e5ad0345c6f2b2138e4fae4afc638cdb7cc68b6070da"
Oct 30 06:26:30.079265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:30.078557    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 06:26:30.079265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:30.078622    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 06:26:31.075203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:31.074569    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 06:26:31.075203 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:31.074633    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 06:26:32.074655 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:32.074611    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 06:26:32.075360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:32.075314    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 06:26:32.375069 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:26:32.374979    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:26:33.067285 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.067248    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 06:26:33.067486 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.067316    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 06:26:33.493092 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.493060    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"]
Oct 30 06:26:33.497529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.497508    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.513206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.512811    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:26:33.516324 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.516288    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"]
Oct 30 06:26:33.522517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.522479    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:26:33.557403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.557371    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vftc\" (UniqueName: \"kubernetes.io/projected/6fe51b29-79aa-4107-86cb-ff59e8a590d8-kube-api-access-2vftc\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.557599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.557437    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.557599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.557554    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.658115 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.658075    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.658348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.658134    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.658348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.658277    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vftc\" (UniqueName: \"kubernetes.io/projected/6fe51b29-79aa-4107-86cb-ff59e8a590d8-kube-api-access-2vftc\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.658835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.658762    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.660534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.660503    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.679138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.679114    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vftc\" (UniqueName: \"kubernetes.io/projected/6fe51b29-79aa-4107-86cb-ff59e8a590d8-kube-api-access-2vftc\") pod \"maas-api-revoke-sub-ec843018-8fv8j\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:33.811329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:33.811213    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:34.120592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:34.120564    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"]
Oct 30 06:26:34.123369 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:26:34.123338    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe51b29_79aa_4107_86cb_ff59e8a590d8.slice/crio-aeb65ea5202aceef1f4bcd11f9f038483fcdf37961325b1e33b11842eaaf0679 WatchSource:0}: Error finding container aeb65ea5202aceef1f4bcd11f9f038483fcdf37961325b1e33b11842eaaf0679: Status 404 returned error can't find the container with id aeb65ea5202aceef1f4bcd11f9f038483fcdf37961325b1e33b11842eaaf0679
Oct 30 06:26:34.931394 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:34.931347    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j" event={"ID":"6fe51b29-79aa-4107-86cb-ff59e8a590d8","Type":"ContainerStarted","Data":"7f57bb33be4154a08720981a0356456b36594e9d55f49d3257818bf6400b95c5"}
Oct 30 06:26:34.931826 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:34.931405    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j" event={"ID":"6fe51b29-79aa-4107-86cb-ff59e8a590d8","Type":"ContainerStarted","Data":"aeb65ea5202aceef1f4bcd11f9f038483fcdf37961325b1e33b11842eaaf0679"}
Oct 30 06:26:34.968678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:34.968630    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j" podStartSLOduration=1.968616792 podStartE2EDuration="1.968616792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:34.966440346 +0000 UTC m=+3309.797477563" watchObservedRunningTime="2026-10-30 06:26:34.968616792 +0000 UTC m=+3309.799654006"
Oct 30 06:26:35.067236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:35.067191    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:26:35.067389 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:35.067261    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:26:35.938834 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:35.938797    5250 generic.go:356] "Generic (PLEG): container finished" podID="6fe51b29-79aa-4107-86cb-ff59e8a590d8" containerID="7f57bb33be4154a08720981a0356456b36594e9d55f49d3257818bf6400b95c5" exitCode=0
Oct 30 06:26:35.939278 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:35.938878    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j" event={"ID":"6fe51b29-79aa-4107-86cb-ff59e8a590d8","Type":"ContainerDied","Data":"7f57bb33be4154a08720981a0356456b36594e9d55f49d3257818bf6400b95c5"}
Oct 30 06:26:36.108396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:36.108352    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:26:36.108597 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:36.108419    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:26:37.107030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.107003    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:37.109223 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.109156    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:26:37.109387 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.109234    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:26:37.192134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.192104    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-openshift-ca\") pod \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") "
Oct 30 06:26:37.192390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.192170    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe51b29-79aa-4107-86cb-ff59e8a590d8-kube-api-access-2vftc\" (UniqueName: \"kubernetes.io/projected/6fe51b29-79aa-4107-86cb-ff59e8a590d8-kube-api-access-2vftc\") pod \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") "
Oct 30 06:26:37.192390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.192235    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-xks-ca\") pod \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\" (UID: \"6fe51b29-79aa-4107-86cb-ff59e8a590d8\") "
Oct 30 06:26:37.192644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.192615    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-openshift-ca" pod "6fe51b29-79aa-4107-86cb-ff59e8a590d8" (UID: "6fe51b29-79aa-4107-86cb-ff59e8a590d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:26:37.195042 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.195014    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-xks-ca" pod "6fe51b29-79aa-4107-86cb-ff59e8a590d8" (UID: "6fe51b29-79aa-4107-86cb-ff59e8a590d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:26:37.195163 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.195021    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe51b29-79aa-4107-86cb-ff59e8a590d8-kube-api-access-2vftc" pod "6fe51b29-79aa-4107-86cb-ff59e8a590d8" (UID: "6fe51b29-79aa-4107-86cb-ff59e8a590d8"). InnerVolumeSpecName "kube-api-access-2vftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:26:37.293431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.292918    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vftc\" (UniqueName: \"kubernetes.io/projected/6fe51b29-79aa-4107-86cb-ff59e8a590d8-kube-api-access-2vftc\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:37.293431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.292951    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:37.293431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.292967    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe51b29-79aa-4107-86cb-ff59e8a590d8-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:26:37.937093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.937046    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:26:37.952952 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.952920    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"
Oct 30 06:26:37.953112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.952949    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j" event={"ID":"6fe51b29-79aa-4107-86cb-ff59e8a590d8","Type":"ContainerDied","Data":"aeb65ea5202aceef1f4bcd11f9f038483fcdf37961325b1e33b11842eaaf0679"}
Oct 30 06:26:37.953112 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:37.952998    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeb65ea5202aceef1f4bcd11f9f038483fcdf37961325b1e33b11842eaaf0679"
Oct 30 06:26:38.154338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:38.154301    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:26:38.154769 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:38.154384    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:26:38.351329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:38.351222    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:26:39.196358 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:39.196302    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:26:39.196890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:39.196390    5250 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:26:39.516114 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:39.516026    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:39.516621 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:39.516516    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:26:39.517250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:26:39.517233    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv"
Oct 30 06:27:01.146035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.145995    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"]
Oct 30 06:27:01.146590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.146545    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6fe51b29-79aa-4107-86cb-ff59e8a590d8" containerName="revoke-keys"
Oct 30 06:27:01.146590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.146563    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe51b29-79aa-4107-86cb-ff59e8a590d8" containerName="revoke-keys"
Oct 30 06:27:01.146731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.146655    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe51b29-79aa-4107-86cb-ff59e8a590d8" containerName="revoke-keys"
Oct 30 06:27:01.149961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.149941    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.162429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.162403    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:27:01.172620 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.172585    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:27:01.176035 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.176008    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"]
Oct 30 06:27:01.234254 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.234224    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.234422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.234362    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vrp\" (UniqueName: \"kubernetes.io/projected/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-kube-api-access-v7vrp\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.234474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.234430    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.335790 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.335755    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.335985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.335805    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.335985 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.335904    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vrp\" (UniqueName: \"kubernetes.io/projected/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-kube-api-access-v7vrp\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.336519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.336497    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.338560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.338536    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.357366 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.357338    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vrp\" (UniqueName: \"kubernetes.io/projected/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-kube-api-access-v7vrp\") pod \"maas-api-revoke-sub-046e1343-4lcck\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.462499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.462430    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:01.759269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:01.759167    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"]
Oct 30 06:27:01.762612 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:27:01.762581    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e29fc8_6eee_42f8_a88a_75cc3bd5a009.slice/crio-d8412dff760a023925ec7734521950978eaaa3318a71968e00e96fcfdddd7026 WatchSource:0}: Error finding container d8412dff760a023925ec7734521950978eaaa3318a71968e00e96fcfdddd7026: Status 404 returned error can't find the container with id d8412dff760a023925ec7734521950978eaaa3318a71968e00e96fcfdddd7026
Oct 30 06:27:02.110984 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:02.110955    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck" event={"ID":"02e29fc8-6eee-42f8-a88a-75cc3bd5a009","Type":"ContainerStarted","Data":"7093bd827aceb795a4419697a0d24a6b20f29886ba325cbb08bf5a331748b851"}
Oct 30 06:27:02.111104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:02.111007    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck" event={"ID":"02e29fc8-6eee-42f8-a88a-75cc3bd5a009","Type":"ContainerStarted","Data":"d8412dff760a023925ec7734521950978eaaa3318a71968e00e96fcfdddd7026"}
Oct 30 06:27:02.147370 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:02.147321    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck" podStartSLOduration=1.147307415 podStartE2EDuration="1.147307415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:02.145378451 +0000 UTC m=+3336.976415666" watchObservedRunningTime="2026-10-30 06:27:02.147307415 +0000 UTC m=+3336.978344652"
Oct 30 06:27:02.372839 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:27:02.372756    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:27:03.118371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:03.118289    5250 generic.go:356] "Generic (PLEG): container finished" podID="02e29fc8-6eee-42f8-a88a-75cc3bd5a009" containerID="7093bd827aceb795a4419697a0d24a6b20f29886ba325cbb08bf5a331748b851" exitCode=0
Oct 30 06:27:03.118371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:03.118343    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck" event={"ID":"02e29fc8-6eee-42f8-a88a-75cc3bd5a009","Type":"ContainerDied","Data":"7093bd827aceb795a4419697a0d24a6b20f29886ba325cbb08bf5a331748b851"}
Oct 30 06:27:04.253578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.253549    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:04.359878 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.359847    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-xks-ca\") pod \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") "
Oct 30 06:27:04.360056 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.359945    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-kube-api-access-v7vrp\" (UniqueName: \"kubernetes.io/projected/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-kube-api-access-v7vrp\") pod \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") "
Oct 30 06:27:04.360056 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.359966    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-openshift-ca\") pod \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\" (UID: \"02e29fc8-6eee-42f8-a88a-75cc3bd5a009\") "
Oct 30 06:27:04.360427 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.360403    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-openshift-ca" pod "02e29fc8-6eee-42f8-a88a-75cc3bd5a009" (UID: "02e29fc8-6eee-42f8-a88a-75cc3bd5a009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:27:04.362138 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.362117    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-xks-ca" pod "02e29fc8-6eee-42f8-a88a-75cc3bd5a009" (UID: "02e29fc8-6eee-42f8-a88a-75cc3bd5a009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:27:04.362233 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.362161    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-kube-api-access-v7vrp" pod "02e29fc8-6eee-42f8-a88a-75cc3bd5a009" (UID: "02e29fc8-6eee-42f8-a88a-75cc3bd5a009"). InnerVolumeSpecName "kube-api-access-v7vrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:27:04.460858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.460827    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7vrp\" (UniqueName: \"kubernetes.io/projected/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-kube-api-access-v7vrp\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:04.460858 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.460855    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:04.461053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:04.460868    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e29fc8-6eee-42f8-a88a-75cc3bd5a009-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:05.130519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:05.130488    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"
Oct 30 06:27:05.131165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:05.130523    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck" event={"ID":"02e29fc8-6eee-42f8-a88a-75cc3bd5a009","Type":"ContainerDied","Data":"d8412dff760a023925ec7734521950978eaaa3318a71968e00e96fcfdddd7026"}
Oct 30 06:27:05.131165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:05.130568    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8412dff760a023925ec7734521950978eaaa3318a71968e00e96fcfdddd7026"
Oct 30 06:27:11.483783 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.483740    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"]
Oct 30 06:27:11.484314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.484107    5250 cpu_manager.go:414] "containerMap: removing container" podUID="02e29fc8-6eee-42f8-a88a-75cc3bd5a009" containerName="revoke-keys"
Oct 30 06:27:11.484314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.484119    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e29fc8-6eee-42f8-a88a-75cc3bd5a009" containerName="revoke-keys"
Oct 30 06:27:11.484314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.484208    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e29fc8-6eee-42f8-a88a-75cc3bd5a009" containerName="revoke-keys"
Oct 30 06:27:11.487573 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.487552    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.502071 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.502044    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:27:11.512160 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.512119    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:27:11.514301 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.514279    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"]
Oct 30 06:27:11.623457 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.623408    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.623647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.623552    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.623647 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.623613    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc65b\" (UniqueName: \"kubernetes.io/projected/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-kube-api-access-zc65b\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.724631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.724600    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc65b\" (UniqueName: \"kubernetes.io/projected/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-kube-api-access-zc65b\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.724835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.724688    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.724835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.724767    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.725465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.725446    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.727030 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.727012    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.745503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.745443    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc65b\" (UniqueName: \"kubernetes.io/projected/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-kube-api-access-zc65b\") pod \"maas-api-revoke-sub-d3fba056-x8cwt\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:11.801025 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:11.800993    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:12.101924 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:12.101876    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"]
Oct 30 06:27:12.105325 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:27:12.105283    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd2baa6_5d7d_4664_9afb_f4e821bc938d.slice/crio-5a346f30e7df69e8f69e623ebc31acda892d1f29e73ac0ad83682c1388530477 WatchSource:0}: Error finding container 5a346f30e7df69e8f69e623ebc31acda892d1f29e73ac0ad83682c1388530477: Status 404 returned error can't find the container with id 5a346f30e7df69e8f69e623ebc31acda892d1f29e73ac0ad83682c1388530477
Oct 30 06:27:12.170199 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:12.170142    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt" event={"ID":"1bd2baa6-5d7d-4664-9afb-f4e821bc938d","Type":"ContainerStarted","Data":"5a346f30e7df69e8f69e623ebc31acda892d1f29e73ac0ad83682c1388530477"}
Oct 30 06:27:13.179589 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:13.179495    5250 generic.go:356] "Generic (PLEG): container finished" podID="1bd2baa6-5d7d-4664-9afb-f4e821bc938d" containerID="639adb4ddf879fc4614e3423704c35ec6d3fb7c66ebe2208866cc30dd45a9611" exitCode=0
Oct 30 06:27:13.179966 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:13.179586    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt" event={"ID":"1bd2baa6-5d7d-4664-9afb-f4e821bc938d","Type":"ContainerDied","Data":"639adb4ddf879fc4614e3423704c35ec6d3fb7c66ebe2208866cc30dd45a9611"}
Oct 30 06:27:14.540396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.540359    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:14.654456 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.654424    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-xks-ca\") pod \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") "
Oct 30 06:27:14.654671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.654496    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-openshift-ca\") pod \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") "
Oct 30 06:27:14.654671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.654609    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-kube-api-access-zc65b\" (UniqueName: \"kubernetes.io/projected/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-kube-api-access-zc65b\") pod \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\" (UID: \"1bd2baa6-5d7d-4664-9afb-f4e821bc938d\") "
Oct 30 06:27:14.654846 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.654824    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-openshift-ca" pod "1bd2baa6-5d7d-4664-9afb-f4e821bc938d" (UID: "1bd2baa6-5d7d-4664-9afb-f4e821bc938d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:27:14.657435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.657405    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-xks-ca" pod "1bd2baa6-5d7d-4664-9afb-f4e821bc938d" (UID: "1bd2baa6-5d7d-4664-9afb-f4e821bc938d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:27:14.723469 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.723345    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-kube-api-access-zc65b" pod "1bd2baa6-5d7d-4664-9afb-f4e821bc938d" (UID: "1bd2baa6-5d7d-4664-9afb-f4e821bc938d"). InnerVolumeSpecName "kube-api-access-zc65b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:27:14.755605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.755517    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc65b\" (UniqueName: \"kubernetes.io/projected/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-kube-api-access-zc65b\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:14.755605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.755562    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:14.755605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:14.755578    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd2baa6-5d7d-4664-9afb-f4e821bc938d-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:15.205359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:15.205325    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"
Oct 30 06:27:15.205561 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:15.205361    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt" event={"ID":"1bd2baa6-5d7d-4664-9afb-f4e821bc938d","Type":"ContainerDied","Data":"5a346f30e7df69e8f69e623ebc31acda892d1f29e73ac0ad83682c1388530477"}
Oct 30 06:27:15.205561 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:15.205421    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a346f30e7df69e8f69e623ebc31acda892d1f29e73ac0ad83682c1388530477"
Oct 30 06:27:24.033530 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:24.033497    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"]
Oct 30 06:27:24.043343 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:24.043315    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"]
Oct 30 06:27:24.045642 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:24.045616    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7da157-fn59d"]
Oct 30 06:27:24.047454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:24.047428    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f664c86-l2xxd"]
Oct 30 06:27:25.278097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:25.278063    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbcf479-cb5e-4a22-a613-9cc66b6803e6" path="/var/lib/kubelet/pods/1dbcf479-cb5e-4a22-a613-9cc66b6803e6/volumes"
Oct 30 06:27:25.278498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:25.278479    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1ed1aa-5d0d-44ea-a879-4451d82665dd" path="/var/lib/kubelet/pods/9f1ed1aa-5d0d-44ea-a879-4451d82665dd/volumes"
Oct 30 06:27:32.373164 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:27:32.373122    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:27:33.055058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.055024    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"]
Oct 30 06:27:33.055449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.055436    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1bd2baa6-5d7d-4664-9afb-f4e821bc938d" containerName="revoke-keys"
Oct 30 06:27:33.055503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.055450    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd2baa6-5d7d-4664-9afb-f4e821bc938d" containerName="revoke-keys"
Oct 30 06:27:33.055541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.055509    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd2baa6-5d7d-4664-9afb-f4e821bc938d" containerName="revoke-keys"
Oct 30 06:27:33.058710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.058691    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.059312 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.059274    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.059446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.059332    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.059519 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.059464    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lzb\" (UniqueName: \"kubernetes.io/projected/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-kube-api-access-t8lzb\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.071100 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.071078    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:27:33.081052 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.081022    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:27:33.083821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.083797    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"]
Oct 30 06:27:33.160494 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.160465    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.160665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.160519    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lzb\" (UniqueName: \"kubernetes.io/projected/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-kube-api-access-t8lzb\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.160665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.160557    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.161197 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.161159    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.163027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.163000    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.182126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.182101    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lzb\" (UniqueName: \"kubernetes.io/projected/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-kube-api-access-t8lzb\") pod \"maas-api-revoke-sub-ae594979-mctfc\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.371665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.371585    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:33.670781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:33.670751    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"]
Oct 30 06:27:33.674279 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:27:33.674244    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2d516d_32e7_43cf_b2b4_77ffc109b56d.slice/crio-dd6d12765f08c3a80fb4b0d68c56a3c0856b5e0433f3ce447f92ed008344f85b WatchSource:0}: Error finding container dd6d12765f08c3a80fb4b0d68c56a3c0856b5e0433f3ce447f92ed008344f85b: Status 404 returned error can't find the container with id dd6d12765f08c3a80fb4b0d68c56a3c0856b5e0433f3ce447f92ed008344f85b
Oct 30 06:27:34.354406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:34.354370    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc" event={"ID":"7e2d516d-32e7-43cf-b2b4-77ffc109b56d","Type":"ContainerStarted","Data":"96764205dabb2e737dc6774a5a7c5652b6598209275552d6542e0fa5563f6e43"}
Oct 30 06:27:34.354572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:34.354421    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc" event={"ID":"7e2d516d-32e7-43cf-b2b4-77ffc109b56d","Type":"ContainerStarted","Data":"dd6d12765f08c3a80fb4b0d68c56a3c0856b5e0433f3ce447f92ed008344f85b"}
Oct 30 06:27:34.391972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:34.391920    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc" podStartSLOduration=1.391902387 podStartE2EDuration="1.391902387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:34.389766004 +0000 UTC m=+3369.220803220" watchObservedRunningTime="2026-10-30 06:27:34.391902387 +0000 UTC m=+3369.222939604"
Oct 30 06:27:35.360944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:35.360907    5250 generic.go:356] "Generic (PLEG): container finished" podID="7e2d516d-32e7-43cf-b2b4-77ffc109b56d" containerID="96764205dabb2e737dc6774a5a7c5652b6598209275552d6542e0fa5563f6e43" exitCode=0
Oct 30 06:27:35.361381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:35.360997    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc" event={"ID":"7e2d516d-32e7-43cf-b2b4-77ffc109b56d","Type":"ContainerDied","Data":"96764205dabb2e737dc6774a5a7c5652b6598209275552d6542e0fa5563f6e43"}
Oct 30 06:27:36.488977 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.488951    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:36.587036 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.587006    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-xks-ca\") pod \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") "
Oct 30 06:27:36.587267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.587070    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-kube-api-access-t8lzb\" (UniqueName: \"kubernetes.io/projected/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-kube-api-access-t8lzb\") pod \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") "
Oct 30 06:27:36.587267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.587093    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-openshift-ca\") pod \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\" (UID: \"7e2d516d-32e7-43cf-b2b4-77ffc109b56d\") "
Oct 30 06:27:36.587553 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.587525    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-openshift-ca" pod "7e2d516d-32e7-43cf-b2b4-77ffc109b56d" (UID: "7e2d516d-32e7-43cf-b2b4-77ffc109b56d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:27:36.589454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.589430    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-xks-ca" pod "7e2d516d-32e7-43cf-b2b4-77ffc109b56d" (UID: "7e2d516d-32e7-43cf-b2b4-77ffc109b56d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:27:36.589454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.589447    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-kube-api-access-t8lzb" pod "7e2d516d-32e7-43cf-b2b4-77ffc109b56d" (UID: "7e2d516d-32e7-43cf-b2b4-77ffc109b56d"). InnerVolumeSpecName "kube-api-access-t8lzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:27:36.688086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.688045    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:36.688086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.688083    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8lzb\" (UniqueName: \"kubernetes.io/projected/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-kube-api-access-t8lzb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:36.688086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:36.688093    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d516d-32e7-43cf-b2b4-77ffc109b56d-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:27:37.378529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:37.378491    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc" event={"ID":"7e2d516d-32e7-43cf-b2b4-77ffc109b56d","Type":"ContainerDied","Data":"dd6d12765f08c3a80fb4b0d68c56a3c0856b5e0433f3ce447f92ed008344f85b"}
Oct 30 06:27:37.378703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:37.378541    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd6d12765f08c3a80fb4b0d68c56a3c0856b5e0433f3ce447f92ed008344f85b"
Oct 30 06:27:37.378703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:37.378505    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"
Oct 30 06:27:38.071803 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:38.071770    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:27:38.472689 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:27:38.472647    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:28:02.371969 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:28:02.371930    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:28:12.654342 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:12.654299    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6qzpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:28:12.654823 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:12.654364    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6qzpc" podUID="f236a47e-f79b-4857-9ab1-9711f03815da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:28:18.475425 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:18.475373    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:28:20.223820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:20.223768    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:28:29.728272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:29.726279    5250 scope.go:122] "RemoveContainer" containerID="4eeabc51b4320229ea8a2a2f818a9dd0390bac16a298815f01c21be7a469ab68"
Oct 30 06:28:29.739506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:29.739478    5250 scope.go:122] "RemoveContainer" containerID="4bcf721e0a579eacdba07d95fdcfee1c33dc3453c8e66e7d1bc01b1d0b195f5c"
Oct 30 06:28:29.748066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:29.748041    5250 scope.go:122] "RemoveContainer" containerID="3a9b5349b1523e1ff6c6a573b5f03588e1596f83a74bc31b5aa462d101525ee0"
Oct 30 06:28:29.779228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:29.773534    5250 scope.go:122] "RemoveContainer" containerID="cfe2452154149ecdfa473f18b3d11ee69031cd4e1987bc66e1404ea5e3729a2c"
Oct 30 06:28:29.842280 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:29.842249    5250 scope.go:122] "RemoveContainer" containerID="13e0a8b484d8727d6332407b0e49e051629ae7974d5ab9567cc67fc77e3fb56b"
Oct 30 06:28:30.044152 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:30.044063    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"]
Oct 30 06:28:30.054140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:30.054103    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a137847f-d5g29"]
Oct 30 06:28:31.279311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:31.279282    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c27c2b-6cdb-4260-b468-d8cb300d7800" path="/var/lib/kubelet/pods/36c27c2b-6cdb-4260-b468-d8cb300d7800/volumes"
Oct 30 06:28:32.373476 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:28:32.373438    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:28:38.189929 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:28:38.189847    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:02.372964 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:29:02.372909    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:29:07.033770 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:07.033736    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"]
Oct 30 06:29:07.043518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:07.043489    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b49ce39-qz2fm"]
Oct 30 06:29:07.279305 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:07.279273    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104b7ef7-e9bc-4f7f-8201-76e815ca69f3" path="/var/lib/kubelet/pods/104b7ef7-e9bc-4f7f-8201-76e815ca69f3/volumes"
Oct 30 06:29:20.026894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:20.026856    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"]
Oct 30 06:29:20.036526 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:20.036492    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e65ec30-xzzvl"]
Oct 30 06:29:21.278316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:21.278278    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503c6375-f162-4cd9-9748-e3a653399161" path="/var/lib/kubelet/pods/503c6375-f162-4cd9-9748-e3a653399161/volumes"
Oct 30 06:29:28.037639 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:28.037603    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"]
Oct 30 06:29:28.046968 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:28.046931    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e382c51-m2kxs"]
Oct 30 06:29:29.278606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:29.278572    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311098d1-3cd9-476d-9f9c-3b57a69da5a6" path="/var/lib/kubelet/pods/311098d1-3cd9-476d-9f9c-3b57a69da5a6/volumes"
Oct 30 06:29:29.681123 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:29.681085    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:29.880828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:29.880797    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:29.955288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:29.955216    5250 scope.go:122] "RemoveContainer" containerID="a242f9633cc71319a4fe0663e36140e24453a970014dad46968b0c79075a1101"
Oct 30 06:29:29.964381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:29.964357    5250 scope.go:122] "RemoveContainer" containerID="ea401194053d56af789e5d9baf1652c7d2580aa7c343abf2681e42bc802a4341"
Oct 30 06:29:29.972820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:29.972801    5250 scope.go:122] "RemoveContainer" containerID="b3ffa02275d9d804b1f20a91c4d8bd30e9e8d157ed6ca344b83594499cc6043a"
Oct 30 06:29:32.373213 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:29:32.373161    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:29:38.299116 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:38.299072    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:51.123321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:51.123289    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:51.309429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:51.309388    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:51.464478 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:51.464438    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:51.605544 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:51.605510    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:51.752256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:51.752145    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:54.679069 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:54.679017    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:54.870108 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:54.870076    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:56.318164 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:56.318121    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:56.492409 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:56.492374    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:56.636652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:56.636620    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:29:58.289860 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:29:58.289828    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:00.096676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.096643    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:00.142546 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.142500    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"]
Oct 30 06:30:00.142983 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.142964    5250 cpu_manager.go:414] "containerMap: removing container" podUID="7e2d516d-32e7-43cf-b2b4-77ffc109b56d" containerName="revoke-keys"
Oct 30 06:30:00.143086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.142985    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2d516d-32e7-43cf-b2b4-77ffc109b56d" containerName="revoke-keys"
Oct 30 06:30:00.143086 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.143071    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2d516d-32e7-43cf-b2b4-77ffc109b56d" containerName="revoke-keys"
Oct 30 06:30:00.146167 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.146142    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:00.169049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.169016    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"]
Oct 30 06:30:00.176331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.176310    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:30:00.214004 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.213973    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e892cfda-693f-4e2b-9c76-00652a446864-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889030-tk68m\" (UID: \"e892cfda-693f-4e2b-9c76-00652a446864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:00.214204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.214155    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nczj9\" (UniqueName: \"kubernetes.io/projected/e892cfda-693f-4e2b-9c76-00652a446864-kube-api-access-nczj9\") pod \"maas-api-key-cleanup-29889030-tk68m\" (UID: \"e892cfda-693f-4e2b-9c76-00652a446864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:00.314856 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.314820    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nczj9\" (UniqueName: \"kubernetes.io/projected/e892cfda-693f-4e2b-9c76-00652a446864-kube-api-access-nczj9\") pod \"maas-api-key-cleanup-29889030-tk68m\" (UID: \"e892cfda-693f-4e2b-9c76-00652a446864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:00.315023 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.314862    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e892cfda-693f-4e2b-9c76-00652a446864-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889030-tk68m\" (UID: \"e892cfda-693f-4e2b-9c76-00652a446864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:00.315491 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.315472    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e892cfda-693f-4e2b-9c76-00652a446864-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889030-tk68m\" (UID: \"e892cfda-693f-4e2b-9c76-00652a446864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:00.339626 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.339598    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nczj9\" (UniqueName: \"kubernetes.io/projected/e892cfda-693f-4e2b-9c76-00652a446864-kube-api-access-nczj9\") pod \"maas-api-key-cleanup-29889030-tk68m\" (UID: \"e892cfda-693f-4e2b-9c76-00652a446864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:00.459165 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.459135    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:00.756782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:00.756692    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"]
Oct 30 06:30:00.760328 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:30:00.760300    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode892cfda_693f_4e2b_9c76_00652a446864.slice/crio-d8f637843203d5caff1a433c0eb65f1fd7b1218ad0897d2989e6e5f6d3c9b228 WatchSource:0}: Error finding container d8f637843203d5caff1a433c0eb65f1fd7b1218ad0897d2989e6e5f6d3c9b228: Status 404 returned error can't find the container with id d8f637843203d5caff1a433c0eb65f1fd7b1218ad0897d2989e6e5f6d3c9b228
Oct 30 06:30:01.252380 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:01.252326    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" event={"ID":"e892cfda-693f-4e2b-9c76-00652a446864","Type":"ContainerStarted","Data":"71aa0cab04fa9bf71a2a05b043e885e6ea727d8ff241f043b08f9377cfbd1d53"}
Oct 30 06:30:01.252786 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:01.252401    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" event={"ID":"e892cfda-693f-4e2b-9c76-00652a446864","Type":"ContainerStarted","Data":"d8f637843203d5caff1a433c0eb65f1fd7b1218ad0897d2989e6e5f6d3c9b228"}
Oct 30 06:30:01.288505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:01.288456    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" podStartSLOduration=1.2884414450000001 podStartE2EDuration="1.288441445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:01.287578435 +0000 UTC m=+3516.118615651" watchObservedRunningTime="2026-10-30 06:30:01.288441445 +0000 UTC m=+3516.119478664"
Oct 30 06:30:01.923379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:01.923344    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:02.258699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:02.258609    5250 generic.go:356] "Generic (PLEG): container finished" podID="e892cfda-693f-4e2b-9c76-00652a446864" containerID="71aa0cab04fa9bf71a2a05b043e885e6ea727d8ff241f043b08f9377cfbd1d53" exitCode=60
Oct 30 06:30:02.258699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:02.258671    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" event={"ID":"e892cfda-693f-4e2b-9c76-00652a446864","Type":"ContainerDied","Data":"71aa0cab04fa9bf71a2a05b043e885e6ea727d8ff241f043b08f9377cfbd1d53"}
Oct 30 06:30:02.259303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:02.258992    5250 scope.go:122] "RemoveContainer" containerID="71aa0cab04fa9bf71a2a05b043e885e6ea727d8ff241f043b08f9377cfbd1d53"
Oct 30 06:30:02.374599 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:30:02.374563    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:30:03.273946 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:03.273852    5250 generic.go:356] "Generic (PLEG): container finished" podID="e892cfda-693f-4e2b-9c76-00652a446864" containerID="d9117cb9aa24ab39e558b63fbb1ac63b9ef98d499a03a23a2d4c33f0a26fd757" exitCode=60
Oct 30 06:30:03.274371 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:03.274276    5250 scope.go:122] "RemoveContainer" containerID="d9117cb9aa24ab39e558b63fbb1ac63b9ef98d499a03a23a2d4c33f0a26fd757"
Oct 30 06:30:03.274553 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:30:03.274530    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889030-tk68m_odh-ai-gateway-infra(e892cfda-693f-4e2b-9c76-00652a446864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" podUID="e892cfda-693f-4e2b-9c76-00652a446864"
Oct 30 06:30:03.278454 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:03.278423    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" event={"ID":"e892cfda-693f-4e2b-9c76-00652a446864","Type":"ContainerDied","Data":"d9117cb9aa24ab39e558b63fbb1ac63b9ef98d499a03a23a2d4c33f0a26fd757"}
Oct 30 06:30:03.278610 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:03.278489    5250 scope.go:122] "RemoveContainer" containerID="71aa0cab04fa9bf71a2a05b043e885e6ea727d8ff241f043b08f9377cfbd1d53"
Oct 30 06:30:03.743628 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:03.743588    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:04.280968 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:04.280927    5250 scope.go:122] "RemoveContainer" containerID="d9117cb9aa24ab39e558b63fbb1ac63b9ef98d499a03a23a2d4c33f0a26fd757"
Oct 30 06:30:04.281460 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:30:04.281215    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889030-tk68m_odh-ai-gateway-infra(e892cfda-693f-4e2b-9c76-00652a446864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" podUID="e892cfda-693f-4e2b-9c76-00652a446864"
Oct 30 06:30:05.617963 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:05.617925    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:07.438434 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:07.438394    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:09.315710 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:09.315673    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:09.627897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:09.627811    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:10.245102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:10.245069    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:10.392505 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:10.392468    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:10.552469 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:10.552384    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:11.204844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:11.204805    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:12.138862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.138817    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"]
Oct 30 06:30:12.144652 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.144631    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.159431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.159404    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-627222\"/\"kube-root-ca.crt\""
Oct 30 06:30:12.169272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.169247    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-627222\"/\"openshift-service-ca.crt\""
Oct 30 06:30:12.169414 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.169287    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-627222\"/\"default-dockercfg-bv5gg\""
Oct 30 06:30:12.169414 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.169354    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-627222\"/\"simulator-single-main-627222-kserve-self-signed-certs\""
Oct 30 06:30:12.171609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.171586    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"]
Oct 30 06:30:12.254253 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.254209    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:12.330392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.330358    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4wk\" (UniqueName: \"kubernetes.io/projected/30133dbf-90b9-45c4-acf9-70928f7cf3d2-kube-api-access-bs4wk\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.330392 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.330391    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-home\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.330607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.330424    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-model-cache\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.330607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.330515    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tmp-dir\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.330607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.330554    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tls-certs\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.330607 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.330583    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-dshm\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.431606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.431516    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tls-certs\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.431606 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.431564    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-dshm\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.431863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.431623    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4wk\" (UniqueName: \"kubernetes.io/projected/30133dbf-90b9-45c4-acf9-70928f7cf3d2-kube-api-access-bs4wk\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.431863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.431639    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-home\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.431863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.431671    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-model-cache\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.431863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.431810    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tmp-dir\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.432124 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.432034    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-model-cache\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.432212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.432154    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-home\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.432212 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.432200    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tmp-dir\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.434319 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.434298    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-dshm\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.434436 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.434386    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tls-certs\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.452613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.452586    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4wk\" (UniqueName: \"kubernetes.io/projected/30133dbf-90b9-45c4-acf9-70928f7cf3d2-kube-api-access-bs4wk\") pod \"simulator-single-main-627222-kserve-76cd85d6c9-gzj7h\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") " pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.458455 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.458378    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:12.763155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.763126    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"]
Oct 30 06:30:12.766916 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:30:12.766888    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30133dbf_90b9_45c4_acf9_70928f7cf3d2.slice/crio-dbfe97188fc7909e98339f48f27b4066ace9f85263e1318b1e963c7da15c6ae1 WatchSource:0}: Error finding container dbfe97188fc7909e98339f48f27b4066ace9f85263e1318b1e963c7da15c6ae1: Status 404 returned error can't find the container with id dbfe97188fc7909e98339f48f27b4066ace9f85263e1318b1e963c7da15c6ae1
Oct 30 06:30:12.969776 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:12.969740    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:13.328467 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:13.328423    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h" event={"ID":"30133dbf-90b9-45c4-acf9-70928f7cf3d2","Type":"ContainerStarted","Data":"74483c38f2ebdc15c8cca814f47bb8133e4e6285e402e967fd347f0c3bd75835"}
Oct 30 06:30:13.328871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:13.328474    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h" event={"ID":"30133dbf-90b9-45c4-acf9-70928f7cf3d2","Type":"ContainerStarted","Data":"dbfe97188fc7909e98339f48f27b4066ace9f85263e1318b1e963c7da15c6ae1"}
Oct 30 06:30:13.387010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:13.386953    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h" podStartSLOduration=1.38694057 podStartE2EDuration="1.38694057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:13.384813188 +0000 UTC m=+3528.215850416" watchObservedRunningTime="2026-10-30 06:30:13.38694057 +0000 UTC m=+3528.217977830"
Oct 30 06:30:13.459227 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:13.459197    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:15.127795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:15.127758    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:16.870396 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:16.870344    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:17.274057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:17.274027    5250 scope.go:122] "RemoveContainer" containerID="d9117cb9aa24ab39e558b63fbb1ac63b9ef98d499a03a23a2d4c33f0a26fd757"
Oct 30 06:30:18.032307 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:18.032274    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"]
Oct 30 06:30:18.042229 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:18.042200    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2x58z"]
Oct 30 06:30:18.358584 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:18.358503    5250 generic.go:356] "Generic (PLEG): container finished" podID="e892cfda-693f-4e2b-9c76-00652a446864" containerID="cd9c7245d9139f513548840abbaabcb735e8aedea90b333f45fdd11f97c759dd" exitCode=60
Oct 30 06:30:18.358768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:18.358576    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" event={"ID":"e892cfda-693f-4e2b-9c76-00652a446864","Type":"ContainerDied","Data":"cd9c7245d9139f513548840abbaabcb735e8aedea90b333f45fdd11f97c759dd"}
Oct 30 06:30:18.358768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:18.358669    5250 scope.go:122] "RemoveContainer" containerID="d9117cb9aa24ab39e558b63fbb1ac63b9ef98d499a03a23a2d4c33f0a26fd757"
Oct 30 06:30:18.359045 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:18.359021    5250 scope.go:122] "RemoveContainer" containerID="cd9c7245d9139f513548840abbaabcb735e8aedea90b333f45fdd11f97c759dd"
Oct 30 06:30:18.359336 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:30:18.359314    5250 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889030-tk68m_odh-ai-gateway-infra(e892cfda-693f-4e2b-9c76-00652a446864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" podUID="e892cfda-693f-4e2b-9c76-00652a446864"
Oct 30 06:30:18.548837 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:18.548799    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:19.279168 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.279139    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d2541c-0924-4952-b756-eabe1c81cd83" path="/var/lib/kubelet/pods/61d2541c-0924-4952-b756-eabe1c81cd83/volumes"
Oct 30 06:30:19.403330 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.403296    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"]
Oct 30 06:30:19.544083 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.544026    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:19.591879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.591850    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e892cfda-693f-4e2b-9c76-00652a446864-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e892cfda-693f-4e2b-9c76-00652a446864-maas-api-service-ca\") pod \"e892cfda-693f-4e2b-9c76-00652a446864\" (UID: \"e892cfda-693f-4e2b-9c76-00652a446864\") "
Oct 30 06:30:19.592344 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.592318    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e892cfda-693f-4e2b-9c76-00652a446864-maas-api-service-ca" pod "e892cfda-693f-4e2b-9c76-00652a446864" (UID: "e892cfda-693f-4e2b-9c76-00652a446864"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:30:19.693003 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.692967    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e892cfda-693f-4e2b-9c76-00652a446864-kube-api-access-nczj9\" (UniqueName: \"kubernetes.io/projected/e892cfda-693f-4e2b-9c76-00652a446864-kube-api-access-nczj9\") pod \"e892cfda-693f-4e2b-9c76-00652a446864\" (UID: \"e892cfda-693f-4e2b-9c76-00652a446864\") "
Oct 30 06:30:19.693216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.693116    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e892cfda-693f-4e2b-9c76-00652a446864-maas-api-service-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:30:19.695144 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.695118    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e892cfda-693f-4e2b-9c76-00652a446864-kube-api-access-nczj9" pod "e892cfda-693f-4e2b-9c76-00652a446864" (UID: "e892cfda-693f-4e2b-9c76-00652a446864"). InnerVolumeSpecName "kube-api-access-nczj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:30:19.793604 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:19.793567    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nczj9\" (UniqueName: \"kubernetes.io/projected/e892cfda-693f-4e2b-9c76-00652a446864-kube-api-access-nczj9\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:30:20.313667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:20.313632    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:20.372754 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:20.372721    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"
Oct 30 06:30:20.372947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:20.372723    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m" event={"ID":"e892cfda-693f-4e2b-9c76-00652a446864","Type":"ContainerDied","Data":"d8f637843203d5caff1a433c0eb65f1fd7b1218ad0897d2989e6e5f6d3c9b228"}
Oct 30 06:30:20.372947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:20.372857    5250 scope.go:122] "RemoveContainer" containerID="cd9c7245d9139f513548840abbaabcb735e8aedea90b333f45fdd11f97c759dd"
Oct 30 06:30:20.417196 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:20.417145    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"]
Oct 30 06:30:20.421315 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:20.421284    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-tk68m"]
Oct 30 06:30:21.278515 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:21.278484    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e892cfda-693f-4e2b-9c76-00652a446864" path="/var/lib/kubelet/pods/e892cfda-693f-4e2b-9c76-00652a446864/volumes"
Oct 30 06:30:22.032089 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:22.032035    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:22.459027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:22.458998    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:22.471833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:22.471808    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:23.401954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:23.401923    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:30:23.834331 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:23.834238    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:25.566820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:25.566790    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:27.295127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:27.295089    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:29.163228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:29.163190    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:30.001314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:30.001279    5250 scope.go:122] "RemoveContainer" containerID="66fd3c5bc8f2cf04b7edc983a9d60d255b3b77744a664ac5dc1696308bc349a2"
Oct 30 06:30:30.010271 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:30.010251    5250 scope.go:122] "RemoveContainer" containerID="62888048f2c85cac44ca39472e77c71b6ee747721578e6b57ecc48add2485027"
Oct 30 06:30:30.018480 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:30.018463    5250 scope.go:122] "RemoveContainer" containerID="5b481b04c3e6418043ef7e4fd8213d204b2e4073c14fb9c25f037d347223bdaa"
Oct 30 06:30:30.026333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:30.026313    5250 scope.go:122] "RemoveContainer" containerID="9513b628f964c7951a49262cd738fca76cf188423724259e74207a583abba151"
Oct 30 06:30:30.036771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:30.036751    5250 scope.go:122] "RemoveContainer" containerID="c2d045f2160b1f0071ea4dbfd1f1c7c5e83826ba0d5ead0c664bf5843903e9e5"
Oct 30 06:30:30.045426 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:30.045405    5250 scope.go:122] "RemoveContainer" containerID="16847787e1b2550dff449d52b9038cf468b6fbe4f0bdb86d170bd04dc7116386"
Oct 30 06:30:30.062157 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:30.062127    5250 scope.go:122] "RemoveContainer" containerID="10abd841d9faeedf630985336568470c75ec5146c65e96adf76dbaf11b635237"
Oct 30 06:30:30.895732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:30.895694    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:32.372946 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:30:32.372908    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:30:32.676964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:32.676831    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:32.950883 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:32.950802    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:34.409884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:34.409852    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:35.428988 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.428946    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"]
Oct 30 06:30:35.429524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.429479    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:35.429524 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.429498    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:35.429667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.429532    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:35.429667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.429540    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:35.429667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.429555    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:35.429667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.429563    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:35.429879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.429669    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:35.429879 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.429686    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:35.434894 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.434872    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.451348 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.451325    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"]
Oct 30 06:30:35.458644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.458610    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:30:35.458761 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.458650    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:30:35.513669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.513637    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2g8l\" (UniqueName: \"kubernetes.io/projected/d5eae262-e784-4245-ad77-64e66aba1d07-kube-api-access-t2g8l\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.513827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.513683    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.513827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.513705    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.615022 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.614994    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2g8l\" (UniqueName: \"kubernetes.io/projected/d5eae262-e784-4245-ad77-64e66aba1d07-kube-api-access-t2g8l\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.615159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.615038    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.615159 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.615061    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.615763 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.615743    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.617602 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.617583    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.636441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.636416    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2g8l\" (UniqueName: \"kubernetes.io/projected/d5eae262-e784-4245-ad77-64e66aba1d07-kube-api-access-t2g8l\") pod \"maas-api-revoke-sub-bb0fb7f6-wxhmz\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:35.747492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:35.747401    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:36.047495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:36.047418    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"]
Oct 30 06:30:36.050885 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:30:36.050856    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5eae262_e784_4245_ad77_64e66aba1d07.slice/crio-34e186829b3e2972424e72f348940ac50172cd4d659317b75b66f876d31c4979 WatchSource:0}: Error finding container 34e186829b3e2972424e72f348940ac50172cd4d659317b75b66f876d31c4979: Status 404 returned error can't find the container with id 34e186829b3e2972424e72f348940ac50172cd4d659317b75b66f876d31c4979
Oct 30 06:30:36.167325 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:36.167287    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:36.464335 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:36.464301    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz" event={"ID":"d5eae262-e784-4245-ad77-64e66aba1d07","Type":"ContainerStarted","Data":"e55cd550aa3c317ed14a8b29cfced70832b6d1c0df47c310cc62984afe814dca"}
Oct 30 06:30:36.464680 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:36.464407    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz" event={"ID":"d5eae262-e784-4245-ad77-64e66aba1d07","Type":"ContainerStarted","Data":"34e186829b3e2972424e72f348940ac50172cd4d659317b75b66f876d31c4979"}
Oct 30 06:30:36.504309 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:36.504254    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz" podStartSLOduration=1.504236579 podStartE2EDuration="1.504236579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:36.499730221 +0000 UTC m=+3551.330767437" watchObservedRunningTime="2026-10-30 06:30:36.504236579 +0000 UTC m=+3551.335273790"
Oct 30 06:30:37.471201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:37.471152    5250 generic.go:356] "Generic (PLEG): container finished" podID="d5eae262-e784-4245-ad77-64e66aba1d07" containerID="e55cd550aa3c317ed14a8b29cfced70832b6d1c0df47c310cc62984afe814dca" exitCode=0
Oct 30 06:30:37.471613 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:37.471214    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz" event={"ID":"d5eae262-e784-4245-ad77-64e66aba1d07","Type":"ContainerDied","Data":"e55cd550aa3c317ed14a8b29cfced70832b6d1c0df47c310cc62984afe814dca"}
Oct 30 06:30:37.991674 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:37.991638    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:38.424572 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.424530    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:38.612308 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.612284    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:38.641827 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.641799    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-xks-ca\") pod \"d5eae262-e784-4245-ad77-64e66aba1d07\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") "
Oct 30 06:30:38.641965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.641872    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-openshift-ca\") pod \"d5eae262-e784-4245-ad77-64e66aba1d07\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") "
Oct 30 06:30:38.641965 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.641923    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5eae262-e784-4245-ad77-64e66aba1d07-kube-api-access-t2g8l\" (UniqueName: \"kubernetes.io/projected/d5eae262-e784-4245-ad77-64e66aba1d07-kube-api-access-t2g8l\") pod \"d5eae262-e784-4245-ad77-64e66aba1d07\" (UID: \"d5eae262-e784-4245-ad77-64e66aba1d07\") "
Oct 30 06:30:38.642288 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.642263    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-openshift-ca" pod "d5eae262-e784-4245-ad77-64e66aba1d07" (UID: "d5eae262-e784-4245-ad77-64e66aba1d07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:30:38.643836 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.643815    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-xks-ca" pod "d5eae262-e784-4245-ad77-64e66aba1d07" (UID: "d5eae262-e784-4245-ad77-64e66aba1d07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:30:38.644115 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.644097    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5eae262-e784-4245-ad77-64e66aba1d07-kube-api-access-t2g8l" pod "d5eae262-e784-4245-ad77-64e66aba1d07" (UID: "d5eae262-e784-4245-ad77-64e66aba1d07"). InnerVolumeSpecName "kube-api-access-t2g8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:30:38.742951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.742874    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:30:38.742951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.742900    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae262-e784-4245-ad77-64e66aba1d07-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:30:38.742951 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:38.742911    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2g8l\" (UniqueName: \"kubernetes.io/projected/d5eae262-e784-4245-ad77-64e66aba1d07-kube-api-access-t2g8l\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:30:39.482533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:39.482502    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz" event={"ID":"d5eae262-e784-4245-ad77-64e66aba1d07","Type":"ContainerDied","Data":"34e186829b3e2972424e72f348940ac50172cd4d659317b75b66f876d31c4979"}
Oct 30 06:30:39.482533 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:39.482524    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"
Oct 30 06:30:39.482762 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:39.482551    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e186829b3e2972424e72f348940ac50172cd4d659317b75b66f876d31c4979"
Oct 30 06:30:39.765470 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:39.765440    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:41.458018 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:41.457973    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:43.483256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:43.483219    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:45.140640 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:45.140608    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:46.502731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.502699    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:46.650822 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.650788    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"]
Oct 30 06:30:46.651207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.651173    5250 cpu_manager.go:414] "containerMap: removing container" podUID="d5eae262-e784-4245-ad77-64e66aba1d07" containerName="revoke-keys"
Oct 30 06:30:46.651207 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.651205    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5eae262-e784-4245-ad77-64e66aba1d07" containerName="revoke-keys"
Oct 30 06:30:46.651417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.651265    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e892cfda-693f-4e2b-9c76-00652a446864" containerName="cleanup"
Oct 30 06:30:46.651417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.651276    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5eae262-e784-4245-ad77-64e66aba1d07" containerName="revoke-keys"
Oct 30 06:30:46.654437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.654420    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.666855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.666830    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:30:46.676855 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.676830    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:30:46.679020 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.678998    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"]
Oct 30 06:30:46.708746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.708717    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwvh\" (UniqueName: \"kubernetes.io/projected/34a8a25d-3944-4524-b091-76836a6cd7b9-kube-api-access-2gwvh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.708931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.708769    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.708931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.708803    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.809634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.809541    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.809634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.809613    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwvh\" (UniqueName: \"kubernetes.io/projected/34a8a25d-3944-4524-b091-76836a6cd7b9-kube-api-access-2gwvh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.809902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.809649    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.810377 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.810350    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.812153 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.812127    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.830736 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.830712    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwvh\" (UniqueName: \"kubernetes.io/projected/34a8a25d-3944-4524-b091-76836a6cd7b9-kube-api-access-2gwvh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:46.967094 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:46.967059    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:47.044208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:47.043396    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:47.266318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:47.266280    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"]
Oct 30 06:30:47.269867 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:30:47.269833    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a8a25d_3944_4524_b091_76836a6cd7b9.slice/crio-c1663f67e8316498d4df4183e817fa9c9e535e9654856d7fca301ae135fcba87 WatchSource:0}: Error finding container c1663f67e8316498d4df4183e817fa9c9e535e9654856d7fca301ae135fcba87: Status 404 returned error can't find the container with id c1663f67e8316498d4df4183e817fa9c9e535e9654856d7fca301ae135fcba87
Oct 30 06:30:47.525902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:47.525818    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw" event={"ID":"34a8a25d-3944-4524-b091-76836a6cd7b9","Type":"ContainerStarted","Data":"c1663f67e8316498d4df4183e817fa9c9e535e9654856d7fca301ae135fcba87"}
Oct 30 06:30:48.532881 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:48.532847    5250 generic.go:356] "Generic (PLEG): container finished" podID="34a8a25d-3944-4524-b091-76836a6cd7b9" containerID="ddd540c1e1739cf3ea7ea640ee707a3cb80a4620c4c807e765af5616a1f2ba39" exitCode=0
Oct 30 06:30:48.533321 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:48.532893    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw" event={"ID":"34a8a25d-3944-4524-b091-76836a6cd7b9","Type":"ContainerDied","Data":"ddd540c1e1739cf3ea7ea640ee707a3cb80a4620c4c807e765af5616a1f2ba39"}
Oct 30 06:30:48.867911 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:48.867740    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:49.671224 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.671167    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:49.737206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.737153    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a8a25d-3944-4524-b091-76836a6cd7b9-kube-api-access-2gwvh\" (UniqueName: \"kubernetes.io/projected/34a8a25d-3944-4524-b091-76836a6cd7b9-kube-api-access-2gwvh\") pod \"34a8a25d-3944-4524-b091-76836a6cd7b9\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") "
Oct 30 06:30:49.737417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.737242    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-xks-ca\") pod \"34a8a25d-3944-4524-b091-76836a6cd7b9\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") "
Oct 30 06:30:49.737417 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.737349    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-openshift-ca\") pod \"34a8a25d-3944-4524-b091-76836a6cd7b9\" (UID: \"34a8a25d-3944-4524-b091-76836a6cd7b9\") "
Oct 30 06:30:49.737702 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.737681    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-openshift-ca" pod "34a8a25d-3944-4524-b091-76836a6cd7b9" (UID: "34a8a25d-3944-4524-b091-76836a6cd7b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:30:49.739585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.739562    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a8a25d-3944-4524-b091-76836a6cd7b9-kube-api-access-2gwvh" pod "34a8a25d-3944-4524-b091-76836a6cd7b9" (UID: "34a8a25d-3944-4524-b091-76836a6cd7b9"). InnerVolumeSpecName "kube-api-access-2gwvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:30:49.739585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.739577    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-xks-ca" pod "34a8a25d-3944-4524-b091-76836a6cd7b9" (UID: "34a8a25d-3944-4524-b091-76836a6cd7b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:30:49.839218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.839105    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gwvh\" (UniqueName: \"kubernetes.io/projected/34a8a25d-3944-4524-b091-76836a6cd7b9-kube-api-access-2gwvh\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:30:49.839218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.839146    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:30:49.839218 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:49.839159    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a8a25d-3944-4524-b091-76836a6cd7b9-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:30:50.544272 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:50.544244    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"
Oct 30 06:30:50.544468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:50.544243    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw" event={"ID":"34a8a25d-3944-4524-b091-76836a6cd7b9","Type":"ContainerDied","Data":"c1663f67e8316498d4df4183e817fa9c9e535e9654856d7fca301ae135fcba87"}
Oct 30 06:30:50.544468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:50.544378    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1663f67e8316498d4df4183e817fa9c9e535e9654856d7fca301ae135fcba87"
Oct 30 06:30:50.600534 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:50.600485    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:51.421862 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:51.421824    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"]
Oct 30 06:30:51.533327 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:51.533296    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:52.307129 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:52.307062    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:53.985912 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:53.985853    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:56.000804 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:56.000767    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:57.183390 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:57.183356    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:57.330980 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:57.330936    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:57.544102 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:57.544004    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:30:59.991631 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:30:59.991596    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:00.106739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:00.106700    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"]
Oct 30 06:31:00.110771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:00.110736    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-627222-l86tw"]
Oct 30 06:31:00.134007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:00.133977    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:00.624673 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:00.624639    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:01.278729 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:01.278697    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a8a25d-3944-4524-b091-76836a6cd7b9" path="/var/lib/kubelet/pods/34a8a25d-3944-4524-b091-76836a6cd7b9/volumes"
Oct 30 06:31:02.374055 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:31:02.373899    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:31:06.028084 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.028010    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"]
Oct 30 06:31:06.037386 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.037358    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cn2jx"]
Oct 30 06:31:06.442757 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.442720    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h" podUID="30133dbf-90b9-45c4-acf9-70928f7cf3d2" containerName="main" containerID="cri-o://74483c38f2ebdc15c8cca814f47bb8133e4e6285e402e967fd347f0c3bd75835" gracePeriod=45
Oct 30 06:31:06.565581 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.565543    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:06.630027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.629999    5250 generic.go:356] "Generic (PLEG): container finished" podID="30133dbf-90b9-45c4-acf9-70928f7cf3d2" containerID="74483c38f2ebdc15c8cca814f47bb8133e4e6285e402e967fd347f0c3bd75835" exitCode=0
Oct 30 06:31:06.630214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.630072    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h" event={"ID":"30133dbf-90b9-45c4-acf9-70928f7cf3d2","Type":"ContainerDied","Data":"74483c38f2ebdc15c8cca814f47bb8133e4e6285e402e967fd347f0c3bd75835"}
Oct 30 06:31:06.697050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.696989    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:31:06.708522 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.708491    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:06.786101 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786068    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tmp-dir\") pod \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") "
Oct 30 06:31:06.786360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786108    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-dshm\") pod \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") "
Oct 30 06:31:06.786360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786130    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30133dbf-90b9-45c4-acf9-70928f7cf3d2-kube-api-access-bs4wk\" (UniqueName: \"kubernetes.io/projected/30133dbf-90b9-45c4-acf9-70928f7cf3d2-kube-api-access-bs4wk\") pod \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") "
Oct 30 06:31:06.786360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786225    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tls-certs\") pod \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") "
Oct 30 06:31:06.786360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786264    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-model-cache\") pod \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") "
Oct 30 06:31:06.786360 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786335    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-home\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-home\") pod \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\" (UID: \"30133dbf-90b9-45c4-acf9-70928f7cf3d2\") "
Oct 30 06:31:06.786678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786478    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tmp-dir" pod "30133dbf-90b9-45c4-acf9-70928f7cf3d2" (UID: "30133dbf-90b9-45c4-acf9-70928f7cf3d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:31:06.786678 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786616    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-model-cache" pod "30133dbf-90b9-45c4-acf9-70928f7cf3d2" (UID: "30133dbf-90b9-45c4-acf9-70928f7cf3d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:31:06.786777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786678    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-home" pod "30133dbf-90b9-45c4-acf9-70928f7cf3d2" (UID: "30133dbf-90b9-45c4-acf9-70928f7cf3d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:31:06.786777 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.786709    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:31:06.788667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.788639    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tls-certs" pod "30133dbf-90b9-45c4-acf9-70928f7cf3d2" (UID: "30133dbf-90b9-45c4-acf9-70928f7cf3d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:31:06.788667 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.788655    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30133dbf-90b9-45c4-acf9-70928f7cf3d2-kube-api-access-bs4wk" pod "30133dbf-90b9-45c4-acf9-70928f7cf3d2" (UID: "30133dbf-90b9-45c4-acf9-70928f7cf3d2"). InnerVolumeSpecName "kube-api-access-bs4wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:31:06.788915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.788902    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-dshm" pod "30133dbf-90b9-45c4-acf9-70928f7cf3d2" (UID: "30133dbf-90b9-45c4-acf9-70928f7cf3d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:31:06.890010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.889887    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:31:06.890010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.889933    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4wk\" (UniqueName: \"kubernetes.io/projected/30133dbf-90b9-45c4-acf9-70928f7cf3d2-kube-api-access-bs4wk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:31:06.890010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.889954    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30133dbf-90b9-45c4-acf9-70928f7cf3d2-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:31:06.890010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.889969    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:31:06.890010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.889982    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30133dbf-90b9-45c4-acf9-70928f7cf3d2-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:31:06.911499 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:06.911457    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:07.283034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:07.282994    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0" path="/var/lib/kubelet/pods/be9e7b99-aad7-40e1-a8f5-04b2ccd70ee0/volumes"
Oct 30 06:31:07.636720 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:07.636694    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"
Oct 30 06:31:07.636893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:07.636737    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h" event={"ID":"30133dbf-90b9-45c4-acf9-70928f7cf3d2","Type":"ContainerDied","Data":"dbfe97188fc7909e98339f48f27b4066ace9f85263e1318b1e963c7da15c6ae1"}
Oct 30 06:31:07.636893 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:07.636791    5250 scope.go:122] "RemoveContainer" containerID="74483c38f2ebdc15c8cca814f47bb8133e4e6285e402e967fd347f0c3bd75835"
Oct 30 06:31:07.689538 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:07.689509    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"]
Oct 30 06:31:07.699412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:07.699386    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-627222/simulator-single-main-627222-kserve-76cd85d6c9-gzj7h"]
Oct 30 06:31:09.278865 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:09.278835    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30133dbf-90b9-45c4-acf9-70928f7cf3d2" path="/var/lib/kubelet/pods/30133dbf-90b9-45c4-acf9-70928f7cf3d2/volumes"
Oct 30 06:31:26.027617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:26.027577    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"]
Oct 30 06:31:26.037617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:26.037589    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tczw4"]
Oct 30 06:31:26.055745 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:26.055722    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:31:26.266468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:26.266440    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:31:27.278617 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:27.278581    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024889b7-83fa-44d1-9421-600d8eeddf54" path="/var/lib/kubelet/pods/024889b7-83fa-44d1-9421-600d8eeddf54/volumes"
Oct 30 06:31:30.111250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:30.111215    5250 scope.go:122] "RemoveContainer" containerID="9d83ac37dab0c139271123ee98bd826b10460827bdd1803a794bb301e49f8c5d"
Oct 30 06:31:32.374158 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:31:32.374123    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:31:34.230421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:34.230384    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:35.014868 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:35.014836    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:36.042569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:36.042475    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:38.026726 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:38.026693    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"]
Oct 30 06:31:38.036259 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:38.036232    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8fv8j"]
Oct 30 06:31:38.553031 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:38.552993    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:39.279383 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:39.279350    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe51b29-79aa-4107-86cb-ff59e8a590d8" path="/var/lib/kubelet/pods/6fe51b29-79aa-4107-86cb-ff59e8a590d8/volumes"
Oct 30 06:31:42.005217 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:42.005167    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:31:57.238034 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:31:57.237998    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:32:02.372317 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:32:02.372282    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:32:05.026619 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:05.026577    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"]
Oct 30 06:32:05.036556 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:05.036528    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4lcck"]
Oct 30 06:32:05.278748 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:05.278668    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e29fc8-6eee-42f8-a88a-75cc3bd5a009" path="/var/lib/kubelet/pods/02e29fc8-6eee-42f8-a88a-75cc3bd5a009/volumes"
Oct 30 06:32:15.026722 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:15.026682    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"]
Oct 30 06:32:15.036829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:15.036794    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x8cwt"]
Oct 30 06:32:15.278730 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:15.278651    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd2baa6-5d7d-4664-9afb-f4e821bc938d" path="/var/lib/kubelet/pods/1bd2baa6-5d7d-4664-9afb-f4e821bc938d/volumes"
Oct 30 06:32:30.140542 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:30.140505    5250 scope.go:122] "RemoveContainer" containerID="2ff12653b75df57f3b1addf78c43550b008cb9ef002618d6865d3283238a5301"
Oct 30 06:32:30.149367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:30.149348    5250 scope.go:122] "RemoveContainer" containerID="d082e3dadc8f012ab71edfb6d436b93d14e0b6d32553be60701778a11ad22144"
Oct 30 06:32:32.373059 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:32:32.373014    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:32:37.031521 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:37.031472    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"]
Oct 30 06:32:37.041250 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:37.041216    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mctfc"]
Oct 30 06:32:37.279048 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:37.279016    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2d516d-32e7-43cf-b2b4-77ffc109b56d" path="/var/lib/kubelet/pods/7e2d516d-32e7-43cf-b2b4-77ffc109b56d/volumes"
Oct 30 06:32:38.734686 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:32:38.734648    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:02.372226 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:33:02.372161    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:33:27.101201 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:27.101133    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:28.011162 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:28.011127    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:30.172731 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:30.172696    5250 scope.go:122] "RemoveContainer" containerID="639adb4ddf879fc4614e3423704c35ec6d3fb7c66ebe2208866cc30dd45a9611"
Oct 30 06:33:30.181554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:30.181532    5250 scope.go:122] "RemoveContainer" containerID="7093bd827aceb795a4419697a0d24a6b20f29886ba325cbb08bf5a331748b851"
Oct 30 06:33:30.189446 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:30.189423    5250 scope.go:122] "RemoveContainer" containerID="7f57bb33be4154a08720981a0356456b36594e9d55f49d3257818bf6400b95c5"
Oct 30 06:33:30.976283 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:30.976231    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:31.126854 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:31.126817    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:32.372644 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:33:32.372609    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:33:32.642895 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:32.642855    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:32.820739 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:32.820701    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:32.967898 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:32.967807    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:33.107889 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:33.107844    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:34.608782 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:34.608747    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:36.395578 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:36.395541    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:38.143106 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:38.143074    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:38.784395 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:38.784343    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:39.179743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:39.179699    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:39.892265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:39.892228    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:41.916756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:41.916724    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:43.764043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:43.764003    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:45.605644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:45.605610    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:45.876053 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:45.875950    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:46.480663 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:46.480628    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:46.627443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:46.627388    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:46.778051 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:46.777934    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:47.434063 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:47.434028    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:48.481650 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.481616    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"]
Oct 30 06:33:48.482214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.482157    5250 cpu_manager.go:414] "containerMap: removing container" podUID="30133dbf-90b9-45c4-acf9-70928f7cf3d2" containerName="main"
Oct 30 06:33:48.482214 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.482193    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30133dbf-90b9-45c4-acf9-70928f7cf3d2" containerName="main"
Oct 30 06:33:48.482379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.482225    5250 cpu_manager.go:414] "containerMap: removing container" podUID="34a8a25d-3944-4524-b091-76836a6cd7b9" containerName="revoke-keys"
Oct 30 06:33:48.482379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.482232    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a8a25d-3944-4524-b091-76836a6cd7b9" containerName="revoke-keys"
Oct 30 06:33:48.482379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.482346    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="30133dbf-90b9-45c4-acf9-70928f7cf3d2" containerName="main"
Oct 30 06:33:48.482379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.482359    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a8a25d-3944-4524-b091-76836a6cd7b9" containerName="revoke-keys"
Oct 30 06:33:48.487038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.487011    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.504450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.504421    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"]
Oct 30 06:33:48.512078 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.512044    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-be81b5\"/\"openshift-service-ca.crt\""
Oct 30 06:33:48.512267 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.512252    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-be81b5\"/\"default-dockercfg-4mlwt\""
Oct 30 06:33:48.512406 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.512389    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-be81b5\"/\"simulator-single-main-be81b5-kserve-self-signed-certs\""
Oct 30 06:33:48.512515 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.512501    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-be81b5\"/\"kube-root-ca.crt\""
Oct 30 06:33:48.531066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.531041    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-model-cache\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.531225 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.531074    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-tmp-dir\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.531225 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.531109    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5pj\" (UniqueName: \"kubernetes.io/projected/0b007843-e71b-412b-8b6c-4981aded83f4-kube-api-access-zg5pj\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.531225 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.531164    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b007843-e71b-412b-8b6c-4981aded83f4-tls-certs\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.531357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.531272    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-home\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.531357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.531338    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-dshm\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.569061 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.569028    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:48.632143 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632100    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-home\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.632363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632150    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-dshm\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.632363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632236    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-model-cache\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.632363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632272    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-tmp-dir\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.632363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632308    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5pj\" (UniqueName: \"kubernetes.io/projected/0b007843-e71b-412b-8b6c-4981aded83f4-kube-api-access-zg5pj\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.632363 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632328    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b007843-e71b-412b-8b6c-4981aded83f4-tls-certs\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.632693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632558    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-home\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.632693 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632578    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-model-cache\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.632953 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.632933    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-tmp-dir\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.634869 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.634840    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-dshm\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.635018 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.634937    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b007843-e71b-412b-8b6c-4981aded83f4-tls-certs\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.652915 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.652891    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5pj\" (UniqueName: \"kubernetes.io/projected/0b007843-e71b-412b-8b6c-4981aded83f4-kube-api-access-zg5pj\") pod \"simulator-single-main-be81b5-kserve-5d75ccf869-btnl2\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") " pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:48.802065 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:48.801976    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:49.099916 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:49.099828    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"]
Oct 30 06:33:49.104910 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:33:49.104870    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b007843_e71b_412b_8b6c_4981aded83f4.slice/crio-05f19629fa8952b6121087e51b02f612995fa1f07f9c161bb5e4f7ff5ff439c9 WatchSource:0}: Error finding container 05f19629fa8952b6121087e51b02f612995fa1f07f9c161bb5e4f7ff5ff439c9: Status 404 returned error can't find the container with id 05f19629fa8952b6121087e51b02f612995fa1f07f9c161bb5e4f7ff5ff439c9
Oct 30 06:33:49.305345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:49.305301    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:49.505978 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:49.505939    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2" event={"ID":"0b007843-e71b-412b-8b6c-4981aded83f4","Type":"ContainerStarted","Data":"bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542"}
Oct 30 06:33:49.505978 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:49.505983    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2" event={"ID":"0b007843-e71b-412b-8b6c-4981aded83f4","Type":"ContainerStarted","Data":"05f19629fa8952b6121087e51b02f612995fa1f07f9c161bb5e4f7ff5ff439c9"}
Oct 30 06:33:49.542206 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:49.542129    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2" podStartSLOduration=1.542114589 podStartE2EDuration="1.542114589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:49.541441132 +0000 UTC m=+3744.372478401" watchObservedRunningTime="2026-10-30 06:33:49.542114589 +0000 UTC m=+3744.373151806"
Oct 30 06:33:49.802493 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:49.802404    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:51.254422 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:51.254381    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:53.094489 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:53.094443    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:54.807204 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:54.807149    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:56.530644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:56.530604    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:58.208367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:58.208321    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:33:58.802651 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:58.802615    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:58.815150 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:58.815126    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:59.567932 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:59.567902    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:33:59.867472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:33:59.867387    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:01.571099 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:01.571062    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:02.375607 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:34:02.375568    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:34:03.291571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:03.291532    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:05.104768 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:05.104734    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:06.852438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:06.852401    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:07.516474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.516439    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"]
Oct 30 06:34:07.520088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.520064    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.532208 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.532170    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:34:07.542136 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.542096    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:34:07.544003 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.543977    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"]
Oct 30 06:34:07.599463 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.599433    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zqq\" (UniqueName: \"kubernetes.io/projected/584e7ff3-a093-4470-aadd-51293fa8f0ad-kube-api-access-j4zqq\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.599675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.599484    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.599675 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.599505    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.700068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.700029    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zqq\" (UniqueName: \"kubernetes.io/projected/584e7ff3-a093-4470-aadd-51293fa8f0ad-kube-api-access-j4zqq\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.700299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.700087    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.700299 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.700108    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.700767 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.700745    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.702773 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.702746    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.720828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.720806    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zqq\" (UniqueName: \"kubernetes.io/projected/584e7ff3-a093-4470-aadd-51293fa8f0ad-kube-api-access-j4zqq\") pod \"maas-api-revoke-sub-c814aad1-6bscg\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:07.833472 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:07.833399    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:08.130373 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:08.130290    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"]
Oct 30 06:34:08.133686 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:34:08.133655    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584e7ff3_a093_4470_aadd_51293fa8f0ad.slice/crio-58b528ffed353554e4b8f2ae81266321c1ea15c62ea361aaba38bb3417ef3a8b WatchSource:0}: Error finding container 58b528ffed353554e4b8f2ae81266321c1ea15c62ea361aaba38bb3417ef3a8b: Status 404 returned error can't find the container with id 58b528ffed353554e4b8f2ae81266321c1ea15c62ea361aaba38bb3417ef3a8b
Oct 30 06:34:08.603249 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:08.603204    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg" event={"ID":"584e7ff3-a093-4470-aadd-51293fa8f0ad","Type":"ContainerStarted","Data":"5684106a2af8822ccf21ab52ce00061bb9c0d6f6eb41dfedbc1aa64f85dd8d68"}
Oct 30 06:34:08.603431 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:08.603269    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg" event={"ID":"584e7ff3-a093-4470-aadd-51293fa8f0ad","Type":"ContainerStarted","Data":"58b528ffed353554e4b8f2ae81266321c1ea15c62ea361aaba38bb3417ef3a8b"}
Oct 30 06:34:08.641386 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:08.641328    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg" podStartSLOduration=1.6413105529999998 podStartE2EDuration="1.641310553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:08.639775565 +0000 UTC m=+3763.470812783" watchObservedRunningTime="2026-10-30 06:34:08.641310553 +0000 UTC m=+3763.472347773"
Oct 30 06:34:08.711623 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:08.711597    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:09.609645 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:09.609610    5250 generic.go:356] "Generic (PLEG): container finished" podID="584e7ff3-a093-4470-aadd-51293fa8f0ad" containerID="5684106a2af8822ccf21ab52ce00061bb9c0d6f6eb41dfedbc1aa64f85dd8d68" exitCode=0
Oct 30 06:34:09.610061 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:09.609694    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg" event={"ID":"584e7ff3-a093-4470-aadd-51293fa8f0ad","Type":"ContainerDied","Data":"5684106a2af8822ccf21ab52ce00061bb9c0d6f6eb41dfedbc1aa64f85dd8d68"}
Oct 30 06:34:10.461789 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.461754    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:10.743914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.743885    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:10.829242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.829204    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-xks-ca\") pod \"584e7ff3-a093-4470-aadd-51293fa8f0ad\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") "
Oct 30 06:34:10.829458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.829278    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-openshift-ca\") pod \"584e7ff3-a093-4470-aadd-51293fa8f0ad\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") "
Oct 30 06:34:10.829458 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.829313    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/584e7ff3-a093-4470-aadd-51293fa8f0ad-kube-api-access-j4zqq\" (UniqueName: \"kubernetes.io/projected/584e7ff3-a093-4470-aadd-51293fa8f0ad-kube-api-access-j4zqq\") pod \"584e7ff3-a093-4470-aadd-51293fa8f0ad\" (UID: \"584e7ff3-a093-4470-aadd-51293fa8f0ad\") "
Oct 30 06:34:10.829660 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.829631    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-openshift-ca" pod "584e7ff3-a093-4470-aadd-51293fa8f0ad" (UID: "584e7ff3-a093-4470-aadd-51293fa8f0ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:34:10.831551 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.831527    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-xks-ca" pod "584e7ff3-a093-4470-aadd-51293fa8f0ad" (UID: "584e7ff3-a093-4470-aadd-51293fa8f0ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:34:10.831669 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.831586    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584e7ff3-a093-4470-aadd-51293fa8f0ad-kube-api-access-j4zqq" pod "584e7ff3-a093-4470-aadd-51293fa8f0ad" (UID: "584e7ff3-a093-4470-aadd-51293fa8f0ad"). InnerVolumeSpecName "kube-api-access-j4zqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:34:10.930064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.930029    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:10.930064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.930060    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584e7ff3-a093-4470-aadd-51293fa8f0ad-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:10.930064 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:10.930070    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zqq\" (UniqueName: \"kubernetes.io/projected/584e7ff3-a093-4470-aadd-51293fa8f0ad-kube-api-access-j4zqq\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:11.621256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:11.621164    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"
Oct 30 06:34:11.621256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:11.621171    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg" event={"ID":"584e7ff3-a093-4470-aadd-51293fa8f0ad","Type":"ContainerDied","Data":"58b528ffed353554e4b8f2ae81266321c1ea15c62ea361aaba38bb3417ef3a8b"}
Oct 30 06:34:11.621256 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:11.621238    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b528ffed353554e4b8f2ae81266321c1ea15c62ea361aaba38bb3417ef3a8b"
Oct 30 06:34:12.135812 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:12.135777    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:13.952975 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:13.952940    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:15.698011 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:15.697976    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:17.398359 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:17.398323    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:18.367871 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.367834    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:18.499194 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.499158    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"]
Oct 30 06:34:18.499643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.499528    5250 cpu_manager.go:414] "containerMap: removing container" podUID="584e7ff3-a093-4470-aadd-51293fa8f0ad" containerName="revoke-keys"
Oct 30 06:34:18.499643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.499540    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="584e7ff3-a093-4470-aadd-51293fa8f0ad" containerName="revoke-keys"
Oct 30 06:34:18.499643 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.499628    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="584e7ff3-a093-4470-aadd-51293fa8f0ad" containerName="revoke-keys"
Oct 30 06:34:18.502329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.502309    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.524113 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.524089    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:34:18.524303 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.524152    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:34:18.526646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.526615    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"]
Oct 30 06:34:18.601127 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.601091    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.601336 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.601170    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.601336 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.601245    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5h65\" (UniqueName: \"kubernetes.io/projected/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-kube-api-access-m5h65\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.702139 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.702104    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.702353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.702149    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.702353 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.702200    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5h65\" (UniqueName: \"kubernetes.io/projected/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-kube-api-access-m5h65\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.702867 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.702842    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.704635 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.704614    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.722756 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.722726    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5h65\" (UniqueName: \"kubernetes.io/projected/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-kube-api-access-m5h65\") pod \"maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:18.815671 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:18.815642    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:19.114126 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:19.113990    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"]
Oct 30 06:34:19.120912 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:34:19.120855    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcacd8e1_f42d_4ed5_85ff_8d2d9da39b37.slice/crio-915e4dbb85dc8afd965bcb8dbe755db752fd16320edd1298b833e4080b1a2c4c WatchSource:0}: Error finding container 915e4dbb85dc8afd965bcb8dbe755db752fd16320edd1298b833e4080b1a2c4c: Status 404 returned error can't find the container with id 915e4dbb85dc8afd965bcb8dbe755db752fd16320edd1298b833e4080b1a2c4c
Oct 30 06:34:19.297950 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:19.297915    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:19.662362 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:19.662323    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q" event={"ID":"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37","Type":"ContainerStarted","Data":"653ff39f9ab8e4494e84d8a2e992a1294faa7cccf4d25b63e5ad1f17d601fa46"}
Oct 30 06:34:19.662883 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:19.662396    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q" event={"ID":"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37","Type":"ContainerStarted","Data":"915e4dbb85dc8afd965bcb8dbe755db752fd16320edd1298b833e4080b1a2c4c"}
Oct 30 06:34:19.699345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:19.699277    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q" podStartSLOduration=1.6992581150000001 podStartE2EDuration="1.699258115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:19.697043908 +0000 UTC m=+3774.528081126" watchObservedRunningTime="2026-10-30 06:34:19.699258115 +0000 UTC m=+3774.530295329"
Oct 30 06:34:20.584155 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:20.584117    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:20.669715 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:20.669687    5250 generic.go:356] "Generic (PLEG): container finished" podID="fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37" containerID="653ff39f9ab8e4494e84d8a2e992a1294faa7cccf4d25b63e5ad1f17d601fa46" exitCode=0
Oct 30 06:34:20.670093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:20.669770    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q" event={"ID":"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37","Type":"ContainerDied","Data":"653ff39f9ab8e4494e84d8a2e992a1294faa7cccf4d25b63e5ad1f17d601fa46"}
Oct 30 06:34:21.219027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.218990    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:21.811961 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.811929    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:21.823771 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.823744    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-kube-api-access-m5h65\" (UniqueName: \"kubernetes.io/projected/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-kube-api-access-m5h65\") pod \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") "
Oct 30 06:34:21.823889 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.823809    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-xks-ca\") pod \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") "
Oct 30 06:34:21.826014 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.825989    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-kube-api-access-m5h65" pod "fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37" (UID: "fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37"). InnerVolumeSpecName "kube-api-access-m5h65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:34:21.826149 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.826080    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-xks-ca" pod "fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37" (UID: "fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:34:21.924788 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.924750    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-openshift-ca\") pod \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\" (UID: \"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37\") "
Oct 30 06:34:21.925087 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.925067    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5h65\" (UniqueName: \"kubernetes.io/projected/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-kube-api-access-m5h65\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:21.925216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.925086    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-openshift-ca" pod "fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37" (UID: "fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:34:21.925216 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:21.925094    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:22.025820 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:22.025780    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:22.681875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:22.681839    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"
Oct 30 06:34:22.681875 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:22.681850    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q" event={"ID":"fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37","Type":"ContainerDied","Data":"915e4dbb85dc8afd965bcb8dbe755db752fd16320edd1298b833e4080b1a2c4c"}
Oct 30 06:34:22.682104 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:22.681901    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915e4dbb85dc8afd965bcb8dbe755db752fd16320edd1298b833e4080b1a2c4c"
Oct 30 06:34:22.969492 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:22.969413    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:23.294024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:23.293944    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"]
Oct 30 06:34:23.400010 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:23.399976    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:24.774616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:24.774584    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:26.473719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:26.473565    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:28.196040 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:28.196001    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:28.817262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:28.817231    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:28.982158 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:28.982130    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:29.124699 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:29.124612    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:29.766995 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:29.766964    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:30.215989 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:30.215960    5250 scope.go:122] "RemoveContainer" containerID="96764205dabb2e737dc6774a5a7c5652b6598209275552d6542e0fa5563f6e43"
Oct 30 06:34:30.271800 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:30.271772    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:30.379590 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:30.379556    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"]
Oct 30 06:34:30.390316 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:30.390283    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-be81b5-j278q"]
Oct 30 06:34:30.420828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:30.420780    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:30.931749 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:30.931713    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:31.278752 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:31.278660    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37" path="/var/lib/kubelet/pods/fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37/volumes"
Oct 30 06:34:32.373485 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:34:32.373437    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:34:36.680477 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:36.680394    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:36.828441 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:36.828400    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:37.029692 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:37.029614    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:38.315838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.315791    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2" podUID="0b007843-e71b-412b-8b6c-4981aded83f4" containerName="main" containerID="cri-o://bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542" gracePeriod=45
Oct 30 06:34:38.562684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.562658    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:34:38.666153 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666123    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-home\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-home\") pod \"0b007843-e71b-412b-8b6c-4981aded83f4\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") "
Oct 30 06:34:38.666376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666206    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-model-cache\") pod \"0b007843-e71b-412b-8b6c-4981aded83f4\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") "
Oct 30 06:34:38.666376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666254    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b007843-e71b-412b-8b6c-4981aded83f4-kube-api-access-zg5pj\" (UniqueName: \"kubernetes.io/projected/0b007843-e71b-412b-8b6c-4981aded83f4-kube-api-access-zg5pj\") pod \"0b007843-e71b-412b-8b6c-4981aded83f4\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") "
Oct 30 06:34:38.666376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666332    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b007843-e71b-412b-8b6c-4981aded83f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b007843-e71b-412b-8b6c-4981aded83f4-tls-certs\") pod \"0b007843-e71b-412b-8b6c-4981aded83f4\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") "
Oct 30 06:34:38.666560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666407    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-tmp-dir\") pod \"0b007843-e71b-412b-8b6c-4981aded83f4\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") "
Oct 30 06:34:38.666560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666445    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-dshm\") pod \"0b007843-e71b-412b-8b6c-4981aded83f4\" (UID: \"0b007843-e71b-412b-8b6c-4981aded83f4\") "
Oct 30 06:34:38.666560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666464    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-model-cache" pod "0b007843-e71b-412b-8b6c-4981aded83f4" (UID: "0b007843-e71b-412b-8b6c-4981aded83f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:34:38.666560 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666474    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-home" pod "0b007843-e71b-412b-8b6c-4981aded83f4" (UID: "0b007843-e71b-412b-8b6c-4981aded83f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:34:38.666798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666699    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-tmp-dir" pod "0b007843-e71b-412b-8b6c-4981aded83f4" (UID: "0b007843-e71b-412b-8b6c-4981aded83f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:34:38.666798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666731    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:38.666798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.666752    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:38.668902 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.668863    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b007843-e71b-412b-8b6c-4981aded83f4-tls-certs" pod "0b007843-e71b-412b-8b6c-4981aded83f4" (UID: "0b007843-e71b-412b-8b6c-4981aded83f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:34:38.669024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.668895    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b007843-e71b-412b-8b6c-4981aded83f4-kube-api-access-zg5pj" pod "0b007843-e71b-412b-8b6c-4981aded83f4" (UID: "0b007843-e71b-412b-8b6c-4981aded83f4"). InnerVolumeSpecName "kube-api-access-zg5pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:34:38.669024 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.668949    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-dshm" pod "0b007843-e71b-412b-8b6c-4981aded83f4" (UID: "0b007843-e71b-412b-8b6c-4981aded83f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:34:38.767403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.767371    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:38.767403 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.767401    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b007843-e71b-412b-8b6c-4981aded83f4-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:38.767615 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.767412    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg5pj\" (UniqueName: \"kubernetes.io/projected/0b007843-e71b-412b-8b6c-4981aded83f4-kube-api-access-zg5pj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:38.767615 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.767422    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b007843-e71b-412b-8b6c-4981aded83f4-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:34:38.771416 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.771387    5250 generic.go:356] "Generic (PLEG): container finished" podID="0b007843-e71b-412b-8b6c-4981aded83f4" containerID="bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542" exitCode=0
Oct 30 06:34:38.771554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.771441    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"
Oct 30 06:34:38.771554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.771468    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2" event={"ID":"0b007843-e71b-412b-8b6c-4981aded83f4","Type":"ContainerDied","Data":"bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542"}
Oct 30 06:34:38.771554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.771514    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2" event={"ID":"0b007843-e71b-412b-8b6c-4981aded83f4","Type":"ContainerDied","Data":"05f19629fa8952b6121087e51b02f612995fa1f07f9c161bb5e4f7ff5ff439c9"}
Oct 30 06:34:38.771554 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.771531    5250 scope.go:122] "RemoveContainer" containerID="bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542"
Oct 30 06:34:38.781058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.781039    5250 scope.go:122] "RemoveContainer" containerID="bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542"
Oct 30 06:34:38.781353 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:34:38.781333    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542\": container with ID starting with bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542 not found: ID does not exist" containerID="bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542"
Oct 30 06:34:38.781412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.781361    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542"} err="failed to get container status \"bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542\": rpc error: code = NotFound desc = could not find container \"bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542\": container with ID starting with bcc7c0e30d0e6903a97e302c4041c3d06e22a2e993467c763f52db51d71c2542 not found: ID does not exist"
Oct 30 06:34:38.815033 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.815004    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"]
Oct 30 06:34:38.817559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.817534    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2"]
Oct 30 06:34:38.895130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:38.895098    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:39.279767 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:39.279731    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b007843-e71b-412b-8b6c-4981aded83f4" path="/var/lib/kubelet/pods/0b007843-e71b-412b-8b6c-4981aded83f4/volumes"
Oct 30 06:34:39.311066 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:39.311040    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:39.557895 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:39.557820    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-be81b5/simulator-single-main-be81b5-kserve-5d75ccf869-btnl2" podUID="0b007843-e71b-412b-8b6c-4981aded83f4" containerName="main" probeResult="failure" output="Get \"https://10.133.1.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:34:42.464863 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:42.464834    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:57.388269 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.388235    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"]
Oct 30 06:34:57.388706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.388612    5250 cpu_manager.go:414] "containerMap: removing container" podUID="fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37" containerName="revoke-keys"
Oct 30 06:34:57.388706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.388628    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37" containerName="revoke-keys"
Oct 30 06:34:57.388706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.388646    5250 cpu_manager.go:414] "containerMap: removing container" podUID="0b007843-e71b-412b-8b6c-4981aded83f4" containerName="main"
Oct 30 06:34:57.388706 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.388651    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b007843-e71b-412b-8b6c-4981aded83f4" containerName="main"
Oct 30 06:34:57.388947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.388739    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcacd8e1-f42d-4ed5-85ff-8d2d9da39b37" containerName="revoke-keys"
Oct 30 06:34:57.388947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.388752    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b007843-e71b-412b-8b6c-4981aded83f4" containerName="main"
Oct 30 06:34:57.393393 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.393366    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.409541 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.409507    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 06:34:57.416481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.416446    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 06:34:57.418329 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.418303    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"]
Oct 30 06:34:57.419646 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.419625    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b658b769-kserve-self-signed-certs\""
Oct 30 06:34:57.419743 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.419627    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bgqg5\""
Oct 30 06:34:57.428437 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.428409    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-dshm\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.428566 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.428471    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cee006e-682e-401b-9cf4-f8c37f472b83-tls-certs\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.428638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.428562    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-tmp-dir\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.428711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.428637    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-model-cache\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.428711 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.428672    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-home\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.428828 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.428715    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhs8\" (UniqueName: \"kubernetes.io/projected/6cee006e-682e-401b-9cf4-f8c37f472b83-kube-api-access-vjhs8\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.483298 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.483266    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:57.529337 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529309    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-tmp-dir\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.529506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529354    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-model-cache\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.529506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529375    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-home\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.529506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529408    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhs8\" (UniqueName: \"kubernetes.io/projected/6cee006e-682e-401b-9cf4-f8c37f472b83-kube-api-access-vjhs8\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.529506 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529490    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-dshm\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.529746 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529524    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cee006e-682e-401b-9cf4-f8c37f472b83-tls-certs\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.529830 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529766    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-tmp-dir\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.529892 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529824    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-model-cache\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.529954 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.529930    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-home\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.531947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.531926    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-dshm\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.532193 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.532157    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cee006e-682e-401b-9cf4-f8c37f472b83-tls-certs\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.550583 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.550556    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhs8\" (UniqueName: \"kubernetes.io/projected/6cee006e-682e-401b-9cf4-f8c37f472b83-kube-api-access-vjhs8\") pod \"e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") " pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:57.706843 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:57.706758    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:34:58.008608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:58.008524    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"]
Oct 30 06:34:58.012246 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:34:58.012216    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cee006e_682e_401b_9cf4_f8c37f472b83.slice/crio-a1ce3644ea313915d3d2ba128232f14a9bf4a9307b60811d636bb10cd3fb955d WatchSource:0}: Error finding container a1ce3644ea313915d3d2ba128232f14a9bf4a9307b60811d636bb10cd3fb955d: Status 404 returned error can't find the container with id a1ce3644ea313915d3d2ba128232f14a9bf4a9307b60811d636bb10cd3fb955d
Oct 30 06:34:58.884665 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:58.884629    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn" event={"ID":"6cee006e-682e-401b-9cf4-f8c37f472b83","Type":"ContainerStarted","Data":"a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6"}
Oct 30 06:34:58.885038 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:58.884673    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn" event={"ID":"6cee006e-682e-401b-9cf4-f8c37f472b83","Type":"ContainerStarted","Data":"a1ce3644ea313915d3d2ba128232f14a9bf4a9307b60811d636bb10cd3fb955d"}
Oct 30 06:34:58.922481 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:58.922428    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn" podStartSLOduration=1.922411275 podStartE2EDuration="1.922411275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:58.920416664 +0000 UTC m=+3813.751453881" watchObservedRunningTime="2026-10-30 06:34:58.922411275 +0000 UTC m=+3813.753448495"
Oct 30 06:34:59.665351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:59.665308    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:34:59.707239 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:34:59.707202    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:35:01.123049 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:01.123014    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:35:02.436683 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:35:02.436645    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:35:07.202684 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:07.202646    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:35:07.706904 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:07.706843    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:35:07.719072 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:07.719046    5250 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:35:07.962251 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:07.962159    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:35:14.543517 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:14.543483    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:35:14.691536 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:14.691505    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:35:28.448849 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:28.448809    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:35:30.657969 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:30.655818    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"]
Oct 30 06:35:30.745074 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:30.745040    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:35:32.437522 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:35:32.437477    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:35:39.026173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:39.026132    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"]
Oct 30 06:35:39.027365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:39.027333    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:35:39.035471 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:39.035446    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb0fb7f6-wxhmz"]
Oct 30 06:35:39.279236 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:39.279141    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5eae262-e784-4245-ad77-64e66aba1d07" path="/var/lib/kubelet/pods/d5eae262-e784-4245-ad77-64e66aba1d07/volumes"
Oct 30 06:35:39.431449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:39.431416    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:35:45.695688 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:45.695637    5250 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn" podUID="6cee006e-682e-401b-9cf4-f8c37f472b83" containerName="main" containerID="cri-o://a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6" gracePeriod=45
Oct 30 06:35:45.940824 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:45.940800    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:35:46.040376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040285    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cee006e-682e-401b-9cf4-f8c37f472b83-kube-api-access-vjhs8\" (UniqueName: \"kubernetes.io/projected/6cee006e-682e-401b-9cf4-f8c37f472b83-kube-api-access-vjhs8\") pod \"6cee006e-682e-401b-9cf4-f8c37f472b83\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") "
Oct 30 06:35:46.040376 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040343    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-home\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-home\") pod \"6cee006e-682e-401b-9cf4-f8c37f472b83\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") "
Oct 30 06:35:46.040634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040389    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-model-cache\") pod \"6cee006e-682e-401b-9cf4-f8c37f472b83\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") "
Oct 30 06:35:46.040634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040523    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-dshm\") pod \"6cee006e-682e-401b-9cf4-f8c37f472b83\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") "
Oct 30 06:35:46.040634 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040568    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-tmp-dir\") pod \"6cee006e-682e-401b-9cf4-f8c37f472b83\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") "
Oct 30 06:35:46.040798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040641    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cee006e-682e-401b-9cf4-f8c37f472b83-tls-certs\" (UniqueName: \"kubernetes.io/secret/6cee006e-682e-401b-9cf4-f8c37f472b83-tls-certs\") pod \"6cee006e-682e-401b-9cf4-f8c37f472b83\" (UID: \"6cee006e-682e-401b-9cf4-f8c37f472b83\") "
Oct 30 06:35:46.040798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040644    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-model-cache" pod "6cee006e-682e-401b-9cf4-f8c37f472b83" (UID: "6cee006e-682e-401b-9cf4-f8c37f472b83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:35:46.040798 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040672    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-home" pod "6cee006e-682e-401b-9cf4-f8c37f472b83" (UID: "6cee006e-682e-401b-9cf4-f8c37f472b83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:35:46.040942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040874    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-tmp-dir" pod "6cee006e-682e-401b-9cf4-f8c37f472b83" (UID: "6cee006e-682e-401b-9cf4-f8c37f472b83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:35:46.040942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040908    5250 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:35:46.040942 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.040926    5250 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:35:46.042810 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.042783    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-dshm" pod "6cee006e-682e-401b-9cf4-f8c37f472b83" (UID: "6cee006e-682e-401b-9cf4-f8c37f472b83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:35:46.042931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.042888    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cee006e-682e-401b-9cf4-f8c37f472b83-kube-api-access-vjhs8" pod "6cee006e-682e-401b-9cf4-f8c37f472b83" (UID: "6cee006e-682e-401b-9cf4-f8c37f472b83"). InnerVolumeSpecName "kube-api-access-vjhs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:35:46.042931 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.042890    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cee006e-682e-401b-9cf4-f8c37f472b83-tls-certs" pod "6cee006e-682e-401b-9cf4-f8c37f472b83" (UID: "6cee006e-682e-401b-9cf4-f8c37f472b83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:35:46.141603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.141577    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjhs8\" (UniqueName: \"kubernetes.io/projected/6cee006e-682e-401b-9cf4-f8c37f472b83-kube-api-access-vjhs8\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:35:46.141603 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.141599    5250 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:35:46.141795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.141629    5250 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cee006e-682e-401b-9cf4-f8c37f472b83-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:35:46.141795 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.141638    5250 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cee006e-682e-401b-9cf4-f8c37f472b83-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:35:46.192656 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.192624    5250 generic.go:356] "Generic (PLEG): container finished" podID="6cee006e-682e-401b-9cf4-f8c37f472b83" containerID="a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6" exitCode=0
Oct 30 06:35:46.192799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.192672    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"
Oct 30 06:35:46.192799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.192687    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn" event={"ID":"6cee006e-682e-401b-9cf4-f8c37f472b83","Type":"ContainerDied","Data":"a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6"}
Oct 30 06:35:46.192799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.192719    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn" event={"ID":"6cee006e-682e-401b-9cf4-f8c37f472b83","Type":"ContainerDied","Data":"a1ce3644ea313915d3d2ba128232f14a9bf4a9307b60811d636bb10cd3fb955d"}
Oct 30 06:35:46.192799 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.192735    5250 scope.go:122] "RemoveContainer" containerID="a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6"
Oct 30 06:35:46.202681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.202663    5250 scope.go:122] "RemoveContainer" containerID="a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6"
Oct 30 06:35:46.203007 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:35:46.202983    5250 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6\": container with ID starting with a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6 not found: ID does not exist" containerID="a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6"
Oct 30 06:35:46.203090 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.203019    5250 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6"} err="failed to get container status \"a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6\": rpc error: code = NotFound desc = could not find container \"a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6\": container with ID starting with a42127e8db585979f12ca4fdca4d82adaf3970161fb74791761990af346da4b6 not found: ID does not exist"
Oct 30 06:35:46.236130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.236104    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"]
Oct 30 06:35:46.245244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:46.245220    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b658b769-kserve-7b59df7d6f-mxjtn"]
Oct 30 06:35:47.318876 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:35:47.318844    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cee006e-682e-401b-9cf4-f8c37f472b83" path="/var/lib/kubelet/pods/6cee006e-682e-401b-9cf4-f8c37f472b83/volumes"
Oct 30 06:36:02.372830 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:36:02.372782    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:36:26.091474 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:26.091438    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:36:26.300232 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:26.300208    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm44x_51684f59-d0c8-4b0e-8f60-977be1864e8c/ovn-acl-logging/0.log"
Oct 30 06:36:32.373064 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:36:32.373026    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:36:34.146445 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.146409    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"]
Oct 30 06:36:34.147130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.146769    5250 cpu_manager.go:414] "containerMap: removing container" podUID="6cee006e-682e-401b-9cf4-f8c37f472b83" containerName="main"
Oct 30 06:36:34.147130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.146781    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cee006e-682e-401b-9cf4-f8c37f472b83" containerName="main"
Oct 30 06:36:34.147130 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.146861    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cee006e-682e-401b-9cf4-f8c37f472b83" containerName="main"
Oct 30 06:36:34.150028 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.150011    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.156852 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.156756    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.156934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.156845    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.156934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.156889    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrz46\" (UniqueName: \"kubernetes.io/projected/684b5383-e465-43a2-8a4e-07a031338d7b-kube-api-access-mrz46\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.162435 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.162414    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:36:34.172173 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.172148    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:36:34.173904 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.173878    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"]
Oct 30 06:36:34.257449 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.257414    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.257625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.257486    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.257625 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.257517    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrz46\" (UniqueName: \"kubernetes.io/projected/684b5383-e465-43a2-8a4e-07a031338d7b-kube-api-access-mrz46\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.258220 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.258165    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.260015 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.259991    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.278847 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.278820    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrz46\" (UniqueName: \"kubernetes.io/projected/684b5383-e465-43a2-8a4e-07a031338d7b-kube-api-access-mrz46\") pod \"maas-api-revoke-sub-f03629db-t8gmw\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.463351 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.463261    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:34.758964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:34.758877    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"]
Oct 30 06:36:34.763379 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:36:34.763349    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684b5383_e465_43a2_8a4e_07a031338d7b.slice/crio-68389ba0a67dcc7e06fdfffb5a711e0698e38c8341e494552b6ec33ce02f5648 WatchSource:0}: Error finding container 68389ba0a67dcc7e06fdfffb5a711e0698e38c8341e494552b6ec33ce02f5648: Status 404 returned error can't find the container with id 68389ba0a67dcc7e06fdfffb5a711e0698e38c8341e494552b6ec33ce02f5648
Oct 30 06:36:35.525401 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:35.525361    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw" event={"ID":"684b5383-e465-43a2-8a4e-07a031338d7b","Type":"ContainerStarted","Data":"5066a5b880c72ed208ed42a4f23a7ba1f4eb32d0723ce373a56252dafd56b450"}
Oct 30 06:36:35.525808 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:35.525424    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw" event={"ID":"684b5383-e465-43a2-8a4e-07a031338d7b","Type":"ContainerStarted","Data":"68389ba0a67dcc7e06fdfffb5a711e0698e38c8341e494552b6ec33ce02f5648"}
Oct 30 06:36:35.562156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:35.562102    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw" podStartSLOduration=1.562087563 podStartE2EDuration="1.562087563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:35.5605823 +0000 UTC m=+3910.391619517" watchObservedRunningTime="2026-10-30 06:36:35.562087563 +0000 UTC m=+3910.393124776"
Oct 30 06:36:36.531873 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:36.531839    5250 generic.go:356] "Generic (PLEG): container finished" podID="684b5383-e465-43a2-8a4e-07a031338d7b" containerID="5066a5b880c72ed208ed42a4f23a7ba1f4eb32d0723ce373a56252dafd56b450" exitCode=0
Oct 30 06:36:36.532367 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:36.531932    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw" event={"ID":"684b5383-e465-43a2-8a4e-07a031338d7b","Type":"ContainerDied","Data":"5066a5b880c72ed208ed42a4f23a7ba1f4eb32d0723ce373a56252dafd56b450"}
Oct 30 06:36:37.670026 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.670003    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:37.679097 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.679078    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684b5383-e465-43a2-8a4e-07a031338d7b-kube-api-access-mrz46\" (UniqueName: \"kubernetes.io/projected/684b5383-e465-43a2-8a4e-07a031338d7b-kube-api-access-mrz46\") pod \"684b5383-e465-43a2-8a4e-07a031338d7b\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") "
Oct 30 06:36:37.679193 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.679148    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-xks-ca\") pod \"684b5383-e465-43a2-8a4e-07a031338d7b\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") "
Oct 30 06:36:37.679244 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.679200    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-openshift-ca\") pod \"684b5383-e465-43a2-8a4e-07a031338d7b\" (UID: \"684b5383-e465-43a2-8a4e-07a031338d7b\") "
Oct 30 06:36:37.679571 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.679551    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-openshift-ca" pod "684b5383-e465-43a2-8a4e-07a031338d7b" (UID: "684b5383-e465-43a2-8a4e-07a031338d7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:36:37.681357 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.681333    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684b5383-e465-43a2-8a4e-07a031338d7b-kube-api-access-mrz46" pod "684b5383-e465-43a2-8a4e-07a031338d7b" (UID: "684b5383-e465-43a2-8a4e-07a031338d7b"). InnerVolumeSpecName "kube-api-access-mrz46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:36:37.681438 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.681336    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-xks-ca" pod "684b5383-e465-43a2-8a4e-07a031338d7b" (UID: "684b5383-e465-43a2-8a4e-07a031338d7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:36:37.779884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.779851    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:36:37.779884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.779878    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684b5383-e465-43a2-8a4e-07a031338d7b-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:36:37.779884 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:37.779887    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrz46\" (UniqueName: \"kubernetes.io/projected/684b5383-e465-43a2-8a4e-07a031338d7b-kube-api-access-mrz46\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:36:38.542616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:38.542584    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw"
Oct 30 06:36:38.542616 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:38.542593    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t8gmw" event={"ID":"684b5383-e465-43a2-8a4e-07a031338d7b","Type":"ContainerDied","Data":"68389ba0a67dcc7e06fdfffb5a711e0698e38c8341e494552b6ec33ce02f5648"}
Oct 30 06:36:38.542844 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:38.542643    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68389ba0a67dcc7e06fdfffb5a711e0698e38c8341e494552b6ec33ce02f5648"
Oct 30 06:36:39.161194 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:39.161141    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:36:39.552314 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:39.552222    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:36:46.885379 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.885343    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"]
Oct 30 06:36:46.885887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.885776    5250 cpu_manager.go:414] "containerMap: removing container" podUID="684b5383-e465-43a2-8a4e-07a031338d7b" containerName="revoke-keys"
Oct 30 06:36:46.885887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.885794    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684b5383-e465-43a2-8a4e-07a031338d7b" containerName="revoke-keys"
Oct 30 06:36:46.885887 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.885872    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="684b5383-e465-43a2-8a4e-07a031338d7b" containerName="revoke-keys"
Oct 30 06:36:46.889050 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.889031    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:46.901545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.901508    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:36:46.901690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.901629    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:36:46.906838 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.906806    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"]
Oct 30 06:36:46.957443 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.957401    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgvb\" (UniqueName: \"kubernetes.io/projected/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-kube-api-access-4jgvb\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:46.957666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.957452    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:46.957666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:46.957590    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:47.058690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.058646    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgvb\" (UniqueName: \"kubernetes.io/projected/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-kube-api-access-4jgvb\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:47.058690 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.058696    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:47.058949 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.058723    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:47.059412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.059381    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:47.061311 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.061282    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:47.083242 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.083212    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgvb\" (UniqueName: \"kubernetes.io/projected/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-kube-api-access-4jgvb\") pod \"maas-api-revoke-sub-0165d7d2-rxmtt\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:47.201974 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.201889    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:47.498007 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.497916    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"]
Oct 30 06:36:47.502172 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:36:47.502146    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f8de3d_7f42_4b58_af1d_4aad9cf9cd2d.slice/crio-441143248c995db4dabbdb022332bf144048272a0f2e25cae2dfacaba85e3649 WatchSource:0}: Error finding container 441143248c995db4dabbdb022332bf144048272a0f2e25cae2dfacaba85e3649: Status 404 returned error can't find the container with id 441143248c995db4dabbdb022332bf144048272a0f2e25cae2dfacaba85e3649
Oct 30 06:36:47.614460 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:47.614423    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt" event={"ID":"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d","Type":"ContainerStarted","Data":"441143248c995db4dabbdb022332bf144048272a0f2e25cae2dfacaba85e3649"}
Oct 30 06:36:48.628851 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:48.628761    5250 generic.go:356] "Generic (PLEG): container finished" podID="e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d" containerID="f185469dcc0ad083f54ab6ecd4e74390e2ba6fff0f3ccb9fe70ec4dd7d327fa1" exitCode=0
Oct 30 06:36:48.629265 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:48.628852    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt" event={"ID":"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d","Type":"ContainerDied","Data":"f185469dcc0ad083f54ab6ecd4e74390e2ba6fff0f3ccb9fe70ec4dd7d327fa1"}
Oct 30 06:36:49.758986 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.758960    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:49.783609 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.783581    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-xks-ca\") pod \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") "
Oct 30 06:36:49.783753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.783677    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-openshift-ca\") pod \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") "
Oct 30 06:36:49.783753 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.783716    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-kube-api-access-4jgvb\" (UniqueName: \"kubernetes.io/projected/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-kube-api-access-4jgvb\") pod \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\" (UID: \"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d\") "
Oct 30 06:36:49.784057 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.784037    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-openshift-ca" pod "e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d" (UID: "e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:36:49.785934 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.785907    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-xks-ca" pod "e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d" (UID: "e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:36:49.786027 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.785988    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-kube-api-access-4jgvb" pod "e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d" (UID: "e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d"). InnerVolumeSpecName "kube-api-access-4jgvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:36:49.885156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.885075    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:36:49.885156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.885105    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jgvb\" (UniqueName: \"kubernetes.io/projected/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-kube-api-access-4jgvb\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:36:49.885156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:49.885115    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:36:50.640709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:50.640670    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt" event={"ID":"e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d","Type":"ContainerDied","Data":"441143248c995db4dabbdb022332bf144048272a0f2e25cae2dfacaba85e3649"}
Oct 30 06:36:50.640709 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:50.640702    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rxmtt"
Oct 30 06:36:50.640941 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:50.640721    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="441143248c995db4dabbdb022332bf144048272a0f2e25cae2dfacaba85e3649"
Oct 30 06:36:57.906429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.906396    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"]
Oct 30 06:36:57.907088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.906990    5250 cpu_manager.go:414] "containerMap: removing container" podUID="e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d" containerName="revoke-keys"
Oct 30 06:36:57.907088 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.907013    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d" containerName="revoke-keys"
Oct 30 06:36:57.907261 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.907107    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d" containerName="revoke-keys"
Oct 30 06:36:57.909935 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.909913    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:57.922412 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.922385    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 06:36:57.932144 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.932116    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\""
Oct 30 06:36:57.934404 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.934330    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"]
Oct 30 06:36:57.959461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.959427    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:57.959461 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.959466    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:57.959809 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:57.959589    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7vg2\" (UniqueName: \"kubernetes.io/projected/1fe69a70-3742-438c-bdb4-dcbf73df47b7-kube-api-access-q7vg2\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:58.060821 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.060782    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:58.060999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.060826    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:58.060999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.060885    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7vg2\" (UniqueName: \"kubernetes.io/projected/1fe69a70-3742-438c-bdb4-dcbf73df47b7-kube-api-access-q7vg2\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:58.061644 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.061622    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:58.063429 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.063409    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:58.081835 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.081814    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7vg2\" (UniqueName: \"kubernetes.io/projected/1fe69a70-3742-438c-bdb4-dcbf73df47b7-kube-api-access-q7vg2\") pod \"maas-api-revoke-sub-6b19338f-cvtl4\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:58.223111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.223027    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:36:58.523365 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.523271    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"]
Oct 30 06:36:58.527726 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:36:58.527696    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe69a70_3742_438c_bdb4_dcbf73df47b7.slice/crio-cc02bc0b011e349d864c3d7e608773a65530a344127cde307a9a896ea41ecbed WatchSource:0}: Error finding container cc02bc0b011e349d864c3d7e608773a65530a344127cde307a9a896ea41ecbed: Status 404 returned error can't find the container with id cc02bc0b011e349d864c3d7e608773a65530a344127cde307a9a896ea41ecbed
Oct 30 06:36:58.684228 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:58.684196    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4" event={"ID":"1fe69a70-3742-438c-bdb4-dcbf73df47b7","Type":"ContainerStarted","Data":"cc02bc0b011e349d864c3d7e608773a65530a344127cde307a9a896ea41ecbed"}
Oct 30 06:36:59.690865 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:59.690775    5250 generic.go:356] "Generic (PLEG): container finished" podID="1fe69a70-3742-438c-bdb4-dcbf73df47b7" containerID="52e763d963dd09abb16a8b6a0837c7541797705375e53299ac5aeb7a0bfbfd67" exitCode=0
Oct 30 06:36:59.690865 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:36:59.690812    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4" event={"ID":"1fe69a70-3742-438c-bdb4-dcbf73df47b7","Type":"ContainerDied","Data":"52e763d963dd09abb16a8b6a0837c7541797705375e53299ac5aeb7a0bfbfd67"}
Oct 30 06:37:00.836840 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.836817    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:37:00.889058 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.889025    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-xks-ca\") pod \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") "
Oct 30 06:37:00.889281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.889071    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe69a70-3742-438c-bdb4-dcbf73df47b7-kube-api-access-q7vg2\" (UniqueName: \"kubernetes.io/projected/1fe69a70-3742-438c-bdb4-dcbf73df47b7-kube-api-access-q7vg2\") pod \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") "
Oct 30 06:37:00.889281 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.889159    5250 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-openshift-ca\") pod \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\" (UID: \"1fe69a70-3742-438c-bdb4-dcbf73df47b7\") "
Oct 30 06:37:00.889532 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.889508    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-openshift-ca" pod "1fe69a70-3742-438c-bdb4-dcbf73df47b7" (UID: "1fe69a70-3742-438c-bdb4-dcbf73df47b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:37:00.891333 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.891304    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-xks-ca" pod "1fe69a70-3742-438c-bdb4-dcbf73df47b7" (UID: "1fe69a70-3742-438c-bdb4-dcbf73df47b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:37:00.891465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.891417    5250 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe69a70-3742-438c-bdb4-dcbf73df47b7-kube-api-access-q7vg2" pod "1fe69a70-3742-438c-bdb4-dcbf73df47b7" (UID: "1fe69a70-3742-438c-bdb4-dcbf73df47b7"). InnerVolumeSpecName "kube-api-access-q7vg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:37:00.990224 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.990114    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-openshift-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:37:00.990224 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.990146    5250 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe69a70-3742-438c-bdb4-dcbf73df47b7-maas-api-xks-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:37:00.990224 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:00.990155    5250 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7vg2\" (UniqueName: \"kubernetes.io/projected/1fe69a70-3742-438c-bdb4-dcbf73df47b7-kube-api-access-q7vg2\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\""
Oct 30 06:37:01.701608 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:01.701576    5250 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4"
Oct 30 06:37:01.701781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:01.701576    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvtl4" event={"ID":"1fe69a70-3742-438c-bdb4-dcbf73df47b7","Type":"ContainerDied","Data":"cc02bc0b011e349d864c3d7e608773a65530a344127cde307a9a896ea41ecbed"}
Oct 30 06:37:01.701781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:01.701705    5250 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc02bc0b011e349d864c3d7e608773a65530a344127cde307a9a896ea41ecbed"
Oct 30 06:37:02.373642 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:37:02.373608    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:37:10.571334 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:10.571287    5250 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vp9dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 06:37:10.571972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:10.571937    5250 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vp9dv" podUID="18c212e1-b3df-43b8-b066-9ffb699837ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 06:37:20.283764 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:20.283729    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwdtw_e228b425-837e-4ba9-beb9-17efa76f7b8d/manager/0.log"
Oct 30 06:37:20.795514 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:20.795482    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-4nqxz_961f8fab-46d9-484a-99e4-a0b80a50ef08/manager/0.log"
Oct 30 06:37:21.341778 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:21.341745    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rtcjb_72779e27-398d-40b1-8c12-c53f7ff376fb/server/0.log"
Oct 30 06:37:21.517947 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:21.517912    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-v7p4v_9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66/manager/0.log"
Oct 30 06:37:21.739972 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:21.739930    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-jbpqr_945cda97-a3c0-4488-a5d1-5bc921afb74f/manager/0.log"
Oct 30 06:37:21.977410 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:21.977381    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9_2cba753f-bbea-4075-9a7b-7648d2e13965/copy-manifests/0.log"
Oct 30 06:37:22.010043 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:22.009977    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9_2cba753f-bbea-4075-9a7b-7648d2e13965/manager/0.log"
Oct 30 06:37:23.893341 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:23.893308    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rxmtt_e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d/revoke-keys/0.log"
Oct 30 06:37:24.635999 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:24.635965    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cvtl4_1fe69a70-3742-438c-bdb4-dcbf73df47b7/revoke-keys/0.log"
Oct 30 06:37:24.826278 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:24.826248    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c814aad1-6bscg_584e7ff3-a093-4470-aadd-51293fa8f0ad/revoke-keys/0.log"
Oct 30 06:37:25.374200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:25.374149    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-t8gmw_684b5383-e465-43a2-8a4e-07a031338d7b/revoke-keys/0.log"
Oct 30 06:37:25.567829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:25.567802    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-c789c_0b4e236c-65dc-4595-81b7-008a8f71d7e5/postgres/0.log"
Oct 30 06:37:27.678068 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:27.678036    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-x7mw6_85d307eb-8102-41eb-9eb2-4197a4d4683e/kuadrant-console-plugin/0.log"
Oct 30 06:37:29.172605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:29.172575    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7b67449fb54lwql_db89190c-4734-40d6-a4e3-ab0ca9c35cec/istio-proxy/0.log"
Oct 30 06:37:30.179468 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:30.179429    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-6qzpc_f236a47e-f79b-4857-9ab1-9711f03815da/istio-proxy/0.log"
Oct 30 06:37:30.264897 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:30.264857    5250 scope.go:122] "RemoveContainer" containerID="e55cd550aa3c317ed14a8b29cfced70832b6d1c0df47c310cc62984afe814dca"
Oct 30 06:37:30.273886 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:30.273866    5250 scope.go:122] "RemoveContainer" containerID="ddd540c1e1739cf3ea7ea640ee707a3cb80a4620c4c807e765af5616a1f2ba39"
Oct 30 06:37:31.377016 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:31.376990    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-vp9dv_18c212e1-b3df-43b8-b066-9ffb699837ed/istio-proxy/0.log"
Oct 30 06:37:31.571134 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:31.571101    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-x56kg_19cd4d21-143f-43b5-b230-0b82140a1567/istio-proxy/0.log"
Oct 30 06:37:32.372898 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:37:32.372865    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:37:36.939714 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:36.939641    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:37:39.275557 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:39.275502    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:37:44.383498 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:44.383460    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qwdtw_e228b425-837e-4ba9-beb9-17efa76f7b8d/manager/0.log"
Oct 30 06:37:44.813733 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:44.813653    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-4nqxz_961f8fab-46d9-484a-99e4-a0b80a50ef08/manager/0.log"
Oct 30 06:37:45.265111 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:45.265083    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rtcjb_72779e27-398d-40b1-8c12-c53f7ff376fb/server/0.log"
Oct 30 06:37:45.451322 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:45.451288    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-v7p4v_9290a5ba-e7ac-4eaa-89e5-99a9b56f2c66/manager/0.log"
Oct 30 06:37:45.663210 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:45.663163    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-jbpqr_945cda97-a3c0-4488-a5d1-5bc921afb74f/manager/0.log"
Oct 30 06:37:45.900874 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:45.900848    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9_2cba753f-bbea-4075-9a7b-7648d2e13965/copy-manifests/0.log"
Oct 30 06:37:45.934732 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:45.934654    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-5wjh9_2cba753f-bbea-4075-9a7b-7648d2e13965/manager/0.log"
Oct 30 06:37:47.560695 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:47.560662    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rxmtt_e7f8de3d-7f42-4b58-af1d-4aad9cf9cd2d/revoke-keys/0.log"
Oct 30 06:37:48.292136 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:48.292111    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cvtl4_1fe69a70-3742-438c-bdb4-dcbf73df47b7/revoke-keys/0.log"
Oct 30 06:37:48.479676 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:48.479648    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c814aad1-6bscg_584e7ff3-a093-4470-aadd-51293fa8f0ad/revoke-keys/0.log"
Oct 30 06:37:49.009890 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:49.009863    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-t8gmw_684b5383-e465-43a2-8a4e-07a031338d7b/revoke-keys/0.log"
Oct 30 06:37:49.182813 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:49.182782    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-c789c_0b4e236c-65dc-4595-81b7-008a8f71d7e5/postgres/0.log"
Oct 30 06:37:50.939156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:50.939125    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-x7mw6_85d307eb-8102-41eb-9eb2-4197a4d4683e/kuadrant-console-plugin/0.log"
Oct 30 06:37:52.109545 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:52.109503    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7b67449fb54lwql_db89190c-4734-40d6-a4e3-ab0ca9c35cec/istio-proxy/0.log"
Oct 30 06:37:53.073928 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:53.073897    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-6qzpc_f236a47e-f79b-4857-9ab1-9711f03815da/istio-proxy/0.log"
Oct 30 06:37:54.253622 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:54.253592    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-vp9dv_18c212e1-b3df-43b8-b066-9ffb699837ed/istio-proxy/0.log"
Oct 30 06:37:54.446022 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:37:54.445988    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-x56kg_19cd4d21-143f-43b5-b230-0b82140a1567/istio-proxy/0.log"
Oct 30 06:38:02.373355 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:38:02.373307    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:38:32.373397 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:38:32.373352    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:38:39.403832 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:38:39.403795    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:39:02.372334 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:39:02.372298    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:39:11.037518 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:11.037426    5250 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"]
Oct 30 06:39:11.047318 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:11.047289    5250 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c814aad1-6bscg"]
Oct 30 06:39:11.278681 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:11.278644    5250 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="584e7ff3-a093-4470-aadd-51293fa8f0ad" path="/var/lib/kubelet/pods/584e7ff3-a093-4470-aadd-51293fa8f0ad/volumes"
Oct 30 06:39:12.077465 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:12.077431    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:39:12.226638 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:12.226600    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:39:14.111781 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.111744    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c66wq/must-gather-shzfn"]
Oct 30 06:39:14.112200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.112138    5250 cpu_manager.go:414] "containerMap: removing container" podUID="1fe69a70-3742-438c-bdb4-dcbf73df47b7" containerName="revoke-keys"
Oct 30 06:39:14.112200 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.112149    5250 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe69a70-3742-438c-bdb4-dcbf73df47b7" containerName="revoke-keys"
Oct 30 06:39:14.112296 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.112244    5250 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe69a70-3742-438c-bdb4-dcbf73df47b7" containerName="revoke-keys"
Oct 30 06:39:14.115503 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.115485    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c66wq/must-gather-shzfn"
Oct 30 06:39:14.137695 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.137673    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:39:14.140339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.140319    5250 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c66wq\"/\"default-dockercfg-vlpb2\""
Oct 30 06:39:14.140339 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.140319    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c66wq\"/\"kube-root-ca.crt\""
Oct 30 06:39:14.140529 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.140485    5250 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c66wq\"/\"openshift-service-ca.crt\""
Oct 30 06:39:14.145559 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.145535    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c66wq/must-gather-shzfn"]
Oct 30 06:39:14.276718 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.276687    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5813c57a-4050-474a-9702-5148ebcf0883-must-gather-output\") pod \"must-gather-shzfn\" (UID: \"5813c57a-4050-474a-9702-5148ebcf0883\") " pod="openshift-must-gather-c66wq/must-gather-shzfn"
Oct 30 06:39:14.276914 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.276728    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhwb\" (UniqueName: \"kubernetes.io/projected/5813c57a-4050-474a-9702-5148ebcf0883-kube-api-access-rvhwb\") pod \"must-gather-shzfn\" (UID: \"5813c57a-4050-474a-9702-5148ebcf0883\") " pod="openshift-must-gather-c66wq/must-gather-shzfn"
Oct 30 06:39:14.282421 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.282387    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:39:14.377332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.377250    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5813c57a-4050-474a-9702-5148ebcf0883-must-gather-output\") pod \"must-gather-shzfn\" (UID: \"5813c57a-4050-474a-9702-5148ebcf0883\") " pod="openshift-must-gather-c66wq/must-gather-shzfn"
Oct 30 06:39:14.377332 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.377285    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhwb\" (UniqueName: \"kubernetes.io/projected/5813c57a-4050-474a-9702-5148ebcf0883-kube-api-access-rvhwb\") pod \"must-gather-shzfn\" (UID: \"5813c57a-4050-474a-9702-5148ebcf0883\") " pod="openshift-must-gather-c66wq/must-gather-shzfn"
Oct 30 06:39:14.377605 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.377584    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5813c57a-4050-474a-9702-5148ebcf0883-must-gather-output\") pod \"must-gather-shzfn\" (UID: \"5813c57a-4050-474a-9702-5148ebcf0883\") " pod="openshift-must-gather-c66wq/must-gather-shzfn"
Oct 30 06:39:14.398734 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.398708    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhwb\" (UniqueName: \"kubernetes.io/projected/5813c57a-4050-474a-9702-5148ebcf0883-kube-api-access-rvhwb\") pod \"must-gather-shzfn\" (UID: \"5813c57a-4050-474a-9702-5148ebcf0883\") " pod="openshift-must-gather-c66wq/must-gather-shzfn"
Oct 30 06:39:14.430213 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.430172    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c66wq/must-gather-shzfn"
Oct 30 06:39:14.727765 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.727728    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c66wq/must-gather-shzfn"]
Oct 30 06:39:14.731274 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:39:14.731241    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5813c57a_4050_474a_9702_5148ebcf0883.slice/crio-ad09a51cf9a47825d79ec8f836f0641d97c61fb9af07b2d552437bb858fe198f WatchSource:0}: Error finding container ad09a51cf9a47825d79ec8f836f0641d97c61fb9af07b2d552437bb858fe198f: Status 404 returned error can't find the container with id ad09a51cf9a47825d79ec8f836f0641d97c61fb9af07b2d552437bb858fe198f
Oct 30 06:39:14.733093 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:14.733077    5250 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 06:39:15.352098 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:15.352065    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/must-gather-shzfn" event={"ID":"5813c57a-4050-474a-9702-5148ebcf0883","Type":"ContainerStarted","Data":"ad09a51cf9a47825d79ec8f836f0641d97c61fb9af07b2d552437bb858fe198f"}
Oct 30 06:39:16.358666 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:16.358626    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/must-gather-shzfn" event={"ID":"5813c57a-4050-474a-9702-5148ebcf0883","Type":"ContainerStarted","Data":"e759143651bb3ae73a1f637e74e1b406a2bfbe3f6be5e9817378d1849657c2f6"}
Oct 30 06:39:16.359140 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:16.358676    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/must-gather-shzfn" event={"ID":"5813c57a-4050-474a-9702-5148ebcf0883","Type":"ContainerStarted","Data":"5c64bf9117e36b39892efcc77f2cb47862f45eded5f8e3bfc523a4ed631e52f2"}
Oct 30 06:39:16.442594 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:16.442540    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c66wq/must-gather-shzfn" podStartSLOduration=1.621887581 podStartE2EDuration="2.442523138s" totalImagesPullingTime="820.635557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:16.440149592 +0000 UTC m=+4071.271186810" watchObservedRunningTime="2026-10-30 06:39:16.442523138 +0000 UTC m=+4071.273560356"
Oct 30 06:39:18.898194 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:18.898147    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r6plj_a581d4f8-dc97-47d9-a409-bd391c36e67f/global-pull-secret-syncer/0.log"
Oct 30 06:39:19.711381 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:19.711340    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log"
Oct 30 06:39:20.250805 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:20.250774    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-194.ec2.internal_2fd355e41b5bc4e012c326b7e91eb1cb/haproxy/0.log"
Oct 30 06:39:26.769235 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:26.769203    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-x7mw6_85d307eb-8102-41eb-9eb2-4197a4d4683e/kuadrant-console-plugin/0.log"
Oct 30 06:39:28.801495 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.801454    5250 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"]
Oct 30 06:39:28.807964 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.807937    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.825262 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.825235    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"]
Oct 30 06:39:28.871146 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.871103    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-proc\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.871146 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.871142    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-lib-modules\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.871450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.871168    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznkb\" (UniqueName: \"kubernetes.io/projected/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-kube-api-access-qznkb\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.871450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.871306    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-sys\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.871450 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.871351    5250 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-podres\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972384 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972337    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznkb\" (UniqueName: \"kubernetes.io/projected/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-kube-api-access-qznkb\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972429    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-sys\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972459    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-podres\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972585 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972562    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-sys\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972581    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-proc\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972607    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-podres\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972634    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-proc\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972664    5250 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-lib-modules\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.972784 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.972710    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-lib-modules\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:28.994815 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:28.994784    5250 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznkb\" (UniqueName: \"kubernetes.io/projected/0f1b5de3-49c6-4f76-a86a-2cfd021bc083-kube-api-access-qznkb\") pod \"perf-node-gather-daemonset-cx4g4\" (UID: \"0f1b5de3-49c6-4f76-a86a-2cfd021bc083\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:29.123475 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:29.123084    5250 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:29.435833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:29.435716    5250 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"]
Oct 30 06:39:29.448899 ip-10-0-130-194 kubenswrapper[5250]: W1030 06:39:29.448865    5250 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f1b5de3_49c6_4f76_a86a_2cfd021bc083.slice/crio-eb52e8d74ec6e3eb1bf3e08db9f93740b382bc2adaeca909220f627618d55d05 WatchSource:0}: Error finding container eb52e8d74ec6e3eb1bf3e08db9f93740b382bc2adaeca909220f627618d55d05: Status 404 returned error can't find the container with id eb52e8d74ec6e3eb1bf3e08db9f93740b382bc2adaeca909220f627618d55d05
Oct 30 06:39:29.482516 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:29.482487    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4" event={"ID":"0f1b5de3-49c6-4f76-a86a-2cfd021bc083","Type":"ContainerStarted","Data":"eb52e8d74ec6e3eb1bf3e08db9f93740b382bc2adaeca909220f627618d55d05"}
Oct 30 06:39:29.511599 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:29.511555    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mdrmp_5070dc7e-02b2-450d-a4f8-0f616fbabc99/kube-state-metrics/0.log"
Oct 30 06:39:29.623569 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:29.623542    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mdrmp_5070dc7e-02b2-450d-a4f8-0f616fbabc99/kube-rbac-proxy-main/0.log"
Oct 30 06:39:29.739662 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:29.739579    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mdrmp_5070dc7e-02b2-450d-a4f8-0f616fbabc99/kube-rbac-proxy-self/0.log"
Oct 30 06:39:30.102848 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:30.102731    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9h24j_1299b210-215f-44c3-9d45-2fda350dc319/node-exporter/0.log"
Oct 30 06:39:30.220156 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:30.220125    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9h24j_1299b210-215f-44c3-9d45-2fda350dc319/kube-rbac-proxy/0.log"
Oct 30 06:39:30.337592 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:30.337560    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9h24j_1299b210-215f-44c3-9d45-2fda350dc319/init-textfile/0.log"
Oct 30 06:39:30.489490 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:30.489457    5250 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4" event={"ID":"0f1b5de3-49c6-4f76-a86a-2cfd021bc083","Type":"ContainerStarted","Data":"0c691b91a3272c11b74cfa68ad14d2e3c22aac04496ba443d71b4d5efc336d4e"}
Oct 30 06:39:30.489703 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:30.489608    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:30.525345 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:30.525286    5250 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4" podStartSLOduration=2.525273295 podStartE2EDuration="2.525273295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:30.525044086 +0000 UTC m=+4085.356081294" watchObservedRunningTime="2026-10-30 06:39:30.525273295 +0000 UTC m=+4085.356310502"
Oct 30 06:39:31.145483 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:31.145453    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xdvj6_664945e3-2c17-4cdf-a60d-4c274b0c9006/kube-rbac-proxy-main/0.log"
Oct 30 06:39:31.266305 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:31.266250    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xdvj6_664945e3-2c17-4cdf-a60d-4c274b0c9006/kube-rbac-proxy-self/0.log"
Oct 30 06:39:31.383885 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:31.383845    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xdvj6_664945e3-2c17-4cdf-a60d-4c274b0c9006/openshift-state-metrics/0.log"
Oct 30 06:39:32.332719 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:32.332682    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dklsf_883d9833-be84-411b-998e-cea7379b6263/prometheus-operator/0.log"
Oct 30 06:39:32.381432 ip-10-0-130-194 kubenswrapper[5250]: E1030 06:39:32.381379    5250 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gszgm" containerName="extractor"
Oct 30 06:39:32.441600 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:32.441573    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dklsf_883d9833-be84-411b-998e-cea7379b6263/kube-rbac-proxy/0.log"
Oct 30 06:39:36.507944 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:36.507912    5250 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-cx4g4"
Oct 30 06:39:39.528718 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:39.528680    5250 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6nsmc_087e82f9-024f-4bcd-97cf-8b7f928b719a/konnectivity-agent/0.log" limit=1024
Oct 30 06:39:40.866459 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:40.866424    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rvnfl_1e3b6130-4545-4c00-ab54-0c40c7a906bb/dns/0.log"
Oct 30 06:39:40.980512 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:40.980466    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rvnfl_1e3b6130-4545-4c00-ab54-0c40c7a906bb/kube-rbac-proxy/0.log"
Oct 30 06:39:41.274833 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:41.274803    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nlkm2_81290ea5-5539-4ab9-9747-673e98c728d4/dns-node-resolver/0.log"
Oct 30 06:39:41.794829 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:41.794802    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7b67449fb54lwql_db89190c-4734-40d6-a4e3-ab0ca9c35cec/istio-proxy/0.log"
Oct 30 06:39:42.712399 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:42.712367    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v25l9_840883aa-27bd-49c8-a676-1d8336ec90c2/node-ca/0.log"
Oct 30 06:39:44.574338 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:44.574310    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7b67449fb54lwql_db89190c-4734-40d6-a4e3-ab0ca9c35cec/istio-proxy/0.log"
Oct 30 06:39:47.855041 ip-10-0-130-194 kubenswrapper[5250]: I1030 06:39:47.855006    5250 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-6qzpc_f236a47e-f79b-4857-9ab1-9711f03815da/istio-proxy/0.log"